Oct 30 12:29:30.585759 ip-10-0-135-24 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 12:29:30.585771 ip-10-0-135-24 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 12:29:30.585780 ip-10-0-135-24 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 12:29:30.586098 ip-10-0-135-24 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 12:29:40.736294 ip-10-0-135-24 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 12:29:40.736309 ip-10-0-135-24 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d128c281341a495486baddbc02a7acfd --
Oct 30 12:31:33.450582 ip-10-0-135-24 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:31:33.850672 ip-10-0-135-24 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:31:33.850672 ip-10-0-135-24 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:31:33.850672 ip-10-0-135-24 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:31:33.850672 ip-10-0-135-24 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:31:33.854837 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854734    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:33.854837 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854835    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:33.854837 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854839    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:33.854837 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854843    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854846    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854849    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854851    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854854    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854857    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854861    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854864    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854866    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854869    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854871    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854874    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854877    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854879    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854882    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854885    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854894    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854897    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854900    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:33.854959 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854903    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854906    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854909    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854911    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854914    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854917    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854919    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854922    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854925    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854927    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854930    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854932    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854935    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854937    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854940    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854943    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854946    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854948    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854951    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854954    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:33.855425 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854956    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854959    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854961    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854964    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854969    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854974    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854978    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854981    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854984    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854987    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854990    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854994    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.854998    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855002    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855006    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855009    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855012    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855015    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:33.855889 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855018    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855029    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855033    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855036    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855038    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855041    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855044    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855047    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855049    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855052    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855055    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855058    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855061    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855063    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855066    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855069    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855072    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855074    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855077    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855081    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:33.856339 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855084    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855086    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855089    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855092    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855094    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855098    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855101    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855103    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855106    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855109    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855112    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855114    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855118    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855120    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855123    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855125    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855128    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855131    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855133    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855136    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:33.856818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855138    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855141    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855144    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855147    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855150    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855153    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855981    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855989    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855992    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855995    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.855998    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856000    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856004    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856007    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856010    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856013    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856015    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856018    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856021    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856024    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:33.857350 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856027    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856029    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856032    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856056    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856061    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856066    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856069    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856072    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856075    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856078    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856081    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856084    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856086    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856089    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856092    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856095    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856097    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856100    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856103    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:33.857832 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856106    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856108    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856111    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856114    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856116    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856119    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856123    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856126    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856128    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856131    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856133    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856136    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856139    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856141    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856144    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856147    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856150    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856153    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856156    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856158    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:33.858373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856161    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856164    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856166    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856169    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856171    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856174    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856176    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856180    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856184    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856186    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856190    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856193    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856196    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856198    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856219    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856223    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856226    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856229    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856233    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:33.858850 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856239    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856242    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856244    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856247    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856249    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856252    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856255    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856257    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856260    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856263    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856265    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856268    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856271    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856274    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856276    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856279    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856281    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856284    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856287    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856290    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:33.859330 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856292    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856295    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856298    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856300    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856304    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856307    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856309    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856312    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856314    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856317    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856319    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856322    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856326    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856329    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856855    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856861    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856864    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856867    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856870    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856872    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:33.859813 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856875    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856878    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856881    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856884    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856887    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856889    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856892    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856895    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856897    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856900    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856903    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856905    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856911    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856913    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856916    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856919    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856922    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856925    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856927    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856930    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:33.860304 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856933    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856936    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856938    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856941    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856945    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856949    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856951    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856954    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856957    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856959    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856963    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856965    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856968    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856970    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856973    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856977    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856980    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856983    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856985    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:33.860790 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856988    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856990    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856993    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856996    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.856998    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857001    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857004    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857006    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857009    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857012    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857014    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857017    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857019    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857022    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857024    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857027    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857036    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857040    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857043    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857054    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:33.861261 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857057    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857061    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857064    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857067    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857070    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857073    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857076    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857079    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857083    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857086    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857090    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857094    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857097    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857100    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857102    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857105    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857108    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857110    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857113    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:33.861799 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857115    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857117    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857120    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857123    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857126    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857128    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857131    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857134    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857136    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857139    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857143    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857146    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857149    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857157    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857160    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857163    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857166    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857168    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857172    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:33.862257 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857174    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857177    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.857183    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.857973    2570 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.857984    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.857997    2570 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858003    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858008    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858011    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858016    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858020    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858023    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858027    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858030    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858034    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858037    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858040    2570 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858044    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858047    2570 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858050    2570 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858053    2570 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858060    2570 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858063    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858067    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:31:33.862705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858070    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858074    2570 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858078    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858089    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858093    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858096    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858101    2570 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858104    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858107    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858111    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858115    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858119    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858122    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858125    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858129    2570 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858132    2570 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858135    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858140    2570 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858143    2570 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858146    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858149    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858152    2570 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858157    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858160    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858163    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:31:33.863281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858166    2570 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858169    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858173    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858176    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858179    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858182    2570 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858185    2570 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858188    2570 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858192    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858196    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858199    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858221    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858225    2570 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858228    2570 flags.go:64] FLAG: --help="false"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858231    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-135-24.ec2.internal"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858235    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858238    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858241    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858245    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858248    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858251    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858254    2570 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858257    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:31:33.863897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858260    2570 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858263    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858267    2570 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858270    2570 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858273    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858276    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858279    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858282    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858285    2570 flags.go:64] FLAG: --lock-file=""
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858288    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858291    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858294    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858300    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858304    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858307    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858310    2570 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858313    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858317    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858320    2570 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858323    2570 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858336    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858345    2570 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858353    2570 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858356    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858359    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:31:33.864561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858362    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858366    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858369    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858372    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858375    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858383    2570 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858386    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858389    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858392    2570 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858396    2570 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858399    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858402    2570 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858405    2570 flags.go:64] FLAG: --port="10250"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858408    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858411    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d8a3246043739615"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858415    2570 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858418    2570 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858421    2570 flags.go:64] FLAG: --register-node="true"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858424    2570 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858428    2570 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858431    2570 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858434    2570 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858437    2570 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858441    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858444    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858447    2570 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:31:33.865149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858450    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858453    2570 flags.go:64] FLAG: --runonce="false"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858456    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858466    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858470    2570 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858473    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858476    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858479    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858483    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858486    2570 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858489    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858492    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858495    2570 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858499    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858502    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858507    2570 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858510    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858515    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858518    2570 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858522    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858526    2570 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858529    2570 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858532    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858535    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858538    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:31:33.865830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858541    2570 flags.go:64] FLAG: --v="2"
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858546    2570 flags.go:64] FLAG: --version="false"
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858550    2570 flags.go:64] FLAG: --vmodule=""
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858554    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.858558    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858716    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858721    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858724    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858727    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858730    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858733    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858736    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858739    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858741    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858744    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858747    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858749    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858752    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858755    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858757    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858760    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:33.866455 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858762    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858767    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858770    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858773    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858776    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858779    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858781    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858784    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858787    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858789    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858792    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858794    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858798    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858800    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858803    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858805    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858808    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858812    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858815    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:33.866960 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858817    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858822    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858825    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858829    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858834    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858837    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858839    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858842    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858845    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858847    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858851    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858853    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858856    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858859    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858863    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858866    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858869    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858872    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858874    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:33.867444 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858877    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858880    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858882    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858885    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858888    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858890    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858893    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858896    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858898    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858901    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858904    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858907    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858910    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858913    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858916    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858919    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858921    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858924    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858928    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858930    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:33.867898 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858933    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858936    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858939    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858941    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858944    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858946    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858949    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858953    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858956    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858959    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858961    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858964    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858967    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858969    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858972    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858976    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858980    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858983    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858986    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:33.868638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858989    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858992    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858995    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.858998    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859002    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859004    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859007    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859010    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859013    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859015    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859018    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859030    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859034    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.859064    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859102    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859106    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:33.869095 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859110    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859113    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859116    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859118    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859121    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859125    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859128    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859131    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859134    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859136    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859140    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859142    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859145    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859148    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859151    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859153    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859158    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859161    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859164    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:33.869534 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859167    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859171    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859174    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859176    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859179    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859182    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859184    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859187    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859189    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859192    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859195    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859198    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859201    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859215    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859218    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859221    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859224    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859226    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859230    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859233    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:33.870124 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859236    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859238    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859241    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859244    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859246    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859249    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859252    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859254    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859257    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859259    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859262    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859264    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859268    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859275    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859277    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859280    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859282    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859285    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859288    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859290    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:33.870659 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859293    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859296    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859298    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859301    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859304    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859307    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859310    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859312    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859315    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859317    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859323    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859326    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859329    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859332    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859335    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859338    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859341    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859344    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859347    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:33.871144 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859350    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859352    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859355    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859357    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859360    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859362    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859367    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859369    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859372    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859374    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859377    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859380    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859383    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859386    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859388    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859391    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859394    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859397    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859400    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859402    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:33.871649 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859405    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859409    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859411    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859415    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859418    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.859420    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.866247    2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.866271    2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.866942    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.866949    2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867017    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867036    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867040    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867043    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867047    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867052    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867055    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867058    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867061    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:33.872121 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867063    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867066    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867069    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867071    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867074    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867077    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867080    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867083    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867085    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867088    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867091    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867093    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867096    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867099    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867101    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867104    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867107    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867110    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867112    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867116    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:33.872666 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867121    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867124    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867126    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867129    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867132    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867134    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867137    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867139    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867142    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867145    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867147    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867152    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867155    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867158    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867161    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867163    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867166    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867169    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867171    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:33.873128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867174    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867177    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867179    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867182    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867185    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867187    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867190    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867192    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867195    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867198    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867201    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867217    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867221    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867223    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867227    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867230    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867233    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867235    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867239    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:33.873595 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867241    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867245    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867248    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867250    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867253    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867255    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867258    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867261    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867263    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867266    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867268    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867271    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867273    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867276    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867279    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867281    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867284    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867286    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867289    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867291    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:33.874053 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867294    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867297    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867300    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867302    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867306    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867310    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867312    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867315    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867318    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867321    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867324    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867327    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867330    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867332    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867335    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867337    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867340    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867342    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:33.874539 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867345    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.867380    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867530    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867537    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867541    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867544    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867547    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867550    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867553    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867556    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867558    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867561    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867563    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867566    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867569    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:33.874961 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867572    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867574    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867577    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867580    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867582    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867585    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867588    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867591    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867594    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867597    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867600    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867603    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867605    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867608    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867611    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867613    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867616    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867619    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867621    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867624    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:33.875354 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867627    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867629    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867632    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867634    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867637    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867639    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867642    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867645    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867647    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867650    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867652    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867655    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867657    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867660    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867662    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867665    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867668    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867671    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867674    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867676    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:33.875833 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867679    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867682    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867685    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867688    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867691    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867693    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867696    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867699    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867701    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867704    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867706    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867710    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867712    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867715    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867717    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867720    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867722    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867725    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867728    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867730    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:33.876325 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867733    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867736    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867738    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867742    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867745    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867747    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867750    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867753    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867756    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867758    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867761    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867764    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867768    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867772    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867776    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867779    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867782    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867785    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867788    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867790    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:33.876802 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867793    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867796    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867798    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867801    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867804    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867806    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867809    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867811    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867814    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867816    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867819    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867822    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867824    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.867848    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867879    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867883    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:33.877390 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867887    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867890    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867894    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867897    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867899    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867903    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867905    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867908    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867911    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867914    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867917    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867924    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867927    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867930    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867932    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867937    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867940    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867944    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867947    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:33.877777 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867950    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867953    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867956    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867959    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867961    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867964    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867966    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867969    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867975    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867978    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867980    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867983    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867985    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867988    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867991    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867994    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.867997    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868000    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868004    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:33.878373 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868007    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868010    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868013    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868015    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868018    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868021    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868459    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868464    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868467    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868470    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868472    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868477    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868479    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868484    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868487    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868489    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868492    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868495    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868497    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868500    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:33.878844 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868503    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868507    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868510    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868513    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868515    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868518    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868521    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868523    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868526    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868528    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868531    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868534    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868536    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868539    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868542    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868544    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868547    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868550    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868552    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868557    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:33.879363 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868560    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868562    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868565    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868568    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868570    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868574    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868577    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868579    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868582    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868585    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868587    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868590    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868593    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868598    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868600    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868603    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868607    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868610    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868613    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868615    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:33.879879 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868618    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868621    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868623    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868626    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868628    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:33.868631    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.868700    2570 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.871898    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.872852    2570 server.go:1014] "Starting client certificate rotation"
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.872953    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.872993    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.875750    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:31:33.880405 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.876973    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:31:33.886689 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.886670    2570 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:31:33.891337 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.891284    2570 log.go:25] "Validated CRI v1 image API"
Oct 30 12:31:33.892202 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.892166    2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:31:33.894774 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.894744    2570 fs.go:135] Filesystem UUIDs: map[6db28a20-2ee0-425e-837f-7ab8bb9d87f3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8cee4342-7991-4ac6-aac6-088d3c347a82:/dev/nvme0n1p3]
Oct 30 12:31:33.894858 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.894770    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 12:31:33.900708 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.900555    2570 manager.go:217] Machine: {Timestamp:2026-10-30 12:31:33.899258508 +0000 UTC m=+0.330146486 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100550 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f6c7ad217886911c8954043c4229e SystemUUID:ec2f6c7a-d217-8869-11c8-954043c4229e BootID:d128c281-341a-4954-86ba-ddbc02a7acfd Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:7a:d3:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:7a:d3:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:27:99:5b:2f:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:31:33.900708 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.900689    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:31:33.900973 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.900896    2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:31:33.902157 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.902126    2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:31:33.902321 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.902159    2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-24.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:31:33.902381 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.902324    2570 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:31:33.902381 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.902334    2570 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:31:33.902381 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.902367    2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:31:33.902458 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.902387    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:31:33.903276 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.903262    2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:31:33.907122 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.907104    2570 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:31:33.907267 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.907257    2570 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:31:33.907902 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.907892    2570 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:31:33.907932 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.907906    2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:31:33.907932 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.907917    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:31:33.907932 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.907927    2570 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:31:33.908028 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.907962    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:31:33.908361 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.908348    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:31:33.911561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.911537    2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:31:33.916773 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.916750    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:31:33.917630 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.917614    2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:31:33.917729 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.917662    2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:31:33.917887 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.917873    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:31:33.917946 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.917892    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:31:33.917946 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.917902    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:31:33.917946 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.917911    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:31:33.917946 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.917920    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:31:33.917946 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.917929    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:31:33.918441 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.918430    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:31:33.918488 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.918444    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:31:33.918531 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.918513    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:31:33.918531 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.918525    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:31:33.918618 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.918540    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:31:33.918618 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.918555    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:31:33.919739 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.919725    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:31:33.919813 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.919742    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:31:33.923769 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.923749    2570 server.go:1263] "Started kubelet"
Oct 30 12:31:33.923882 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.923839    2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:31:33.923942 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.923838    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:31:33.923942 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.923921    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:31:33.924645 ip-10-0-135-24 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:31:33.926013 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.925997    2570 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:31:33.926099 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.926080    2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:31:33.931897 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.931874    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:31:33.932218 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:33.932182    2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 12:31:33.932346 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.932324    2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:31:33.932649 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:33.932572    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:33.932924 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.932902    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:31:33.932924 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.932905    2570 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:31:33.933048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.932936    2570 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:31:33.933095 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933056    2570 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:31:33.933095 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933065    2570 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:31:33.933424 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933404    2570 factory.go:55] Registering systemd factory
Oct 30 12:31:33.933424 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933425    2570 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:31:33.933659 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933643    2570 factory.go:153] Registering CRI-O factory
Oct 30 12:31:33.933729 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933667    2570 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:31:33.933729 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933717    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:31:33.933823 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933736    2570 factory.go:103] Registering Raw factory
Oct 30 12:31:33.933823 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.933752    2570 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:31:33.934962 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.934945    2570 manager.go:319] Starting recovery of all containers
Oct 30 12:31:33.943875 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.943815    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:31:33.946879 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.946852    2570 manager.go:324] Recovery completed
Oct 30 12:31:33.952271 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.952245    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:33.954228 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.954196    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:33.954344 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.954243    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:33.954344 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.954266    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:33.955036 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.955005    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:33.955149 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.955007    2570 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:31:33.955198 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.955153    2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:31:33.955198 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.955182    2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:31:33.955904 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.955889    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:33.955963 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.955918    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:33.955963 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.955932    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:33.962728 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:33.962699    2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-24.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 12:31:33.967764 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.967738    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:31:33.974260 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.974230    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-24.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 12:31:33.981445 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:33.981408    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 12:31:33.990913 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:33.990880    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-24.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 12:31:33.991846 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:33.990878    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-24.ec2.internal.18e34e9898775f3b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-24.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-24.ec2.internal,},FirstTimestamp:2026-10-30 12:31:33.923544891 +0000 UTC m=+0.354432870,LastTimestamp:2026-10-30 12:31:33.923544891 +0000 UTC m=+0.354432870,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-24.ec2.internal,}"
Oct 30 12:31:33.993281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:33.993263    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2hqd"
Oct 30 12:31:34.011282 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.011251    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2hqd"
Oct 30 12:31:34.013397 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.013374    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 12:31:34.013768 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.013693    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-24.ec2.internal.18e34e989a4b89da  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-24.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-24.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-24.ec2.internal,},FirstTimestamp:2026-10-30 12:31:33.95422665 +0000 UTC m=+0.385114627,LastTimestamp:2026-10-30 12:31:33.95422665 +0000 UTC m=+0.385114627,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-24.ec2.internal,}"
Oct 30 12:31:34.030995 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.030894    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-24.ec2.internal.18e34e989a4bddd9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-24.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-24.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-24.ec2.internal,},FirstTimestamp:2026-10-30 12:31:33.954248153 +0000 UTC m=+0.385136131,LastTimestamp:2026-10-30 12:31:33.954248153 +0000 UTC m=+0.385136131,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-24.ec2.internal,}"
Oct 30 12:31:34.033672 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.033641    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.078358 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.078240    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-24.ec2.internal.18e34e989a4c37e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-24.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-24.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-24.ec2.internal,},FirstTimestamp:2026-10-30 12:31:33.954271204 +0000 UTC m=+0.385159181,LastTimestamp:2026-10-30 12:31:33.954271204 +0000 UTC m=+0.385159181,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-24.ec2.internal,}"
Oct 30 12:31:34.093460 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.093434    2570 policy_none.go:50] "Start"
Oct 30 12:31:34.093460 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.093463    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:31:34.093585 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.093489    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:31:34.095575 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.095560    2570 policy_none.go:44] "Start"
Oct 30 12:31:34.122048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.121853    2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:31:34.126024 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.126010    2570 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:31:34.126155 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.126048    2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:31:34.126155 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.126056    2570 server.go:87] "Starting device plugin registration server"
Oct 30 12:31:34.126325 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.126314    2570 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:31:34.126359 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.126326    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:31:34.126458 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.126433    2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:31:34.126571 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.126505    2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:31:34.126571 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.126514    2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:31:34.127146 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.127125    2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:31:34.127250 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.127187    2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.191421 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.191387    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:31:34.191421 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.191416    2570 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:31:34.193544 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.191440    2570 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:31:34.193544 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.191477    2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 12:31:34.193544 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.191719    2570 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-24.ec2.internal\" not found" node="ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.206440 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.206410    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:34.226942 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.226915    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:34.227881 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.227862    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:34.227987 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.227897    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:34.227987 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.227911    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:34.227987 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.227941    2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.248587 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.248557    2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.248754 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.248588    2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-24.ec2.internal\": node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.292695 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.292648    2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"]
Oct 30 12:31:34.292887 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.292732    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:34.294220 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.294191    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.294864 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.294845    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:34.294953 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.294875    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:34.294953 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.294885    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:34.296950 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.296936    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:34.297085 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.297070    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.297121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.297102    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:34.297810 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.297794    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:34.297810 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.297804    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:34.297919 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.297824    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:34.297919 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.297827    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:34.297919 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.297835    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:34.298002 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.297836    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:34.299913 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.299897    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.299982 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.299924    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:34.300674 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.300659    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:34.300744 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.300685    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:34.300744 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.300696    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:34.321249 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.321198    2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-24.ec2.internal\" not found" node="ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.325243 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.325221    2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-24.ec2.internal\" not found" node="ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.335315 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.335293    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.335422 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.335324    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.335422 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.335349    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdd80fd13fc4b06ccce2af5848f9d942-config\") pod \"kube-apiserver-proxy-ip-10-0-135-24.ec2.internal\" (UID: \"fdd80fd13fc4b06ccce2af5848f9d942\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.395408 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.395319    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.435668 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.435639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdd80fd13fc4b06ccce2af5848f9d942-config\") pod \"kube-apiserver-proxy-ip-10-0-135-24.ec2.internal\" (UID: \"fdd80fd13fc4b06ccce2af5848f9d942\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.435804 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.435671    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.435804 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.435690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.435804 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.435750    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.435804 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.435765    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.435804 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.435768    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdd80fd13fc4b06ccce2af5848f9d942-config\") pod \"kube-apiserver-proxy-ip-10-0-135-24.ec2.internal\" (UID: \"fdd80fd13fc4b06ccce2af5848f9d942\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.496250 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.496176    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.596954 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.596919    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.624506 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.624471    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.628380 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.628357    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:34.697525 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.697420    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.802401 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.802365    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:34.874883 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.874844    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 12:31:34.875444 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:34.875007    2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:31:34.903302 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:34.903270    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:35.004339 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:35.004302    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:31:35.022051 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.022021    2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:35.022342 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.022262    2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:35.033358 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.033328    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:35.067940 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.067907    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:31:35.070198 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.070174    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:31:35.091100 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.091071    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:31:35.097385 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.097335    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:26:34 +0000 UTC" deadline="2028-07-22 19:52:36.571653288 +0000 UTC"
Oct 30 12:31:35.097385 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.097383    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15151h21m1.474273348s"
Oct 30 12:31:35.176128 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:35.176084    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7813451d010c378fca54e5b82e788cc7.slice/crio-9b27de364aa5303a86c83db913102d4c4d79f2b4c36a4f69cf6c8d9e3e22ef31 WatchSource:0}: Error finding container 9b27de364aa5303a86c83db913102d4c4d79f2b4c36a4f69cf6c8d9e3e22ef31: Status 404 returned error can't find the container with id 9b27de364aa5303a86c83db913102d4c4d79f2b4c36a4f69cf6c8d9e3e22ef31
Oct 30 12:31:35.176751 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:35.176726    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd80fd13fc4b06ccce2af5848f9d942.slice/crio-8670e2183233dad6028f9f37b60801aefef0cf2a99a0f93f76cab709b9537bdf WatchSource:0}: Error finding container 8670e2183233dad6028f9f37b60801aefef0cf2a99a0f93f76cab709b9537bdf: Status 404 returned error can't find the container with id 8670e2183233dad6028f9f37b60801aefef0cf2a99a0f93f76cab709b9537bdf
Oct 30 12:31:35.183684 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.183656    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:31:35.195893 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.195826    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal" event={"ID":"7813451d010c378fca54e5b82e788cc7","Type":"ContainerStarted","Data":"9b27de364aa5303a86c83db913102d4c4d79f2b4c36a4f69cf6c8d9e3e22ef31"}
Oct 30 12:31:35.196983 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.196955    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal" event={"ID":"fdd80fd13fc4b06ccce2af5848f9d942","Type":"ContainerStarted","Data":"8670e2183233dad6028f9f37b60801aefef0cf2a99a0f93f76cab709b9537bdf"}
Oct 30 12:31:35.301369 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.301261    2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:35.757127 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.757097    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:35.909365 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.909330    2570 apiserver.go:52] "Watching apiserver"
Oct 30 12:31:35.933484 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.933444    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 12:31:35.942618 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.942589    2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:31:35.942986 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.942963    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp","openshift-cluster-node-tuning-operator/tuned-sq7sj","openshift-network-diagnostics/network-check-target-fkhv4","kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal","openshift-image-registry/node-ca-wjpmw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal","openshift-dns/node-resolver-98w6d","openshift-multus/multus-89m26","openshift-multus/network-metrics-daemon-7dxkx","openshift-ovn-kubernetes/ovnkube-node-svlgw","kube-system/konnectivity-agent-tnrf7","openshift-multus/multus-additional-cni-plugins-n6hfj","openshift-network-operator/iptables-alerter-4mvs6","kube-system/global-pull-secret-syncer-s4gm9"]
Oct 30 12:31:35.945752 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.945723    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:35.948274 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.948248    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:35.951192 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.951166    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:35.951316 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:35.951259    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:35.953796 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.953761    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:35.956569 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.956101    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:35.958894 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.958873    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-89m26"
Oct 30 12:31:35.961824 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.961802    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:35.961943 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:35.961899    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:35.965312 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.965253    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:35.967239 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.967211    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:31:35.967427 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.967411    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxtxl\""
Oct 30 12:31:35.967573 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.967554    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:31:35.967633 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.967589    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:31:35.967823 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.967784    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:31:35.967985 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.967967    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:31:35.970541 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.970357    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:35.970904 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.970868    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k87jk\""
Oct 30 12:31:35.973591 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.973569    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:35.977910 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.977887    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:35.980389 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.980366    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:35.980503 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:35.980449    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:35.993099 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.992088    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:31:35.993099 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:35.992172    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j2lr6"
Oct 30 12:31:36.001873 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.001845    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j2lr6"
Oct 30 12:31:36.034293 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.034258    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:31:36.046047 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046011    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-multus\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.046187 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046059    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd41889-9433-4fdb-8e05-52ea68e32536-agent-certs\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:36.046187 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd41889-9433-4fdb-8e05-52ea68e32536-konnectivity-ca\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:36.046187 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046118    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.046337 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046197    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-node-log\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.046337 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046301    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-netd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.046414 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046334    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovn-node-metrics-cert\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.046414 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046378    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.046502 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.046544 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046518    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cni-binary-copy\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.046605 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046544    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.046678 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046622    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-hostroot\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.046678 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046657    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-etc-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.046817 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-tmp-dir\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.046817 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046767    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ng7\" (UniqueName: \"kubernetes.io/projected/46fbee35-1719-4228-a543-874c0d3f4b4b-kube-api-access-k5ng7\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.046925 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046827    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-var-lib-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.046925 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046870    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85738300-00ba-4ba6-be20-2b259922fd7d-iptables-alerter-script\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.047026 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046926    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:36.047026 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046953    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-conf\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.047026 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.046988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-multus-certs\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.047026 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047010    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-etc-kubernetes\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047035    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047056    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-systemd\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047084    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-var-lib-kubelet\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047111    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-systemd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047135    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-bin\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047160    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-device-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047176    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-netns\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047192    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-daemon-config\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.047226 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047214    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-kubernetes\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047265    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46fbee35-1719-4228-a543-874c0d3f4b4b-serviceca\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047325    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-hosts-file\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047363    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-run\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047377    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-tuned\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047408    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s9mw\" (UniqueName: \"kubernetes.io/projected/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-kube-api-access-8s9mw\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-netns\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-config\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047482    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-registration-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047503    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-script-lib\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047525    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047547    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnb2q\" (UniqueName: \"kubernetes.io/projected/6662ac83-7fc2-451d-b26b-68d934badad1-kube-api-access-nnb2q\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047569    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047594    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047621    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.047694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047644    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-kubelet-config\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cnibin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-socket-dir-parent\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047816    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvpv\" (UniqueName: \"kubernetes.io/projected/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-kube-api-access-vnvpv\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047842    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-ovn\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047868    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdptk\" (UniqueName: \"kubernetes.io/projected/f8924cae-fc43-49bd-9bab-d4bc56220d6a-kube-api-access-sdptk\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047902    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047929    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-k8s-cni-cncf-io\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047957    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-env-overrides\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.047989    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-dbus\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048014    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048039    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-bin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-kubelet\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048093    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdjl\" (UniqueName: \"kubernetes.io/projected/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-kube-api-access-2wdjl\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048118    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-kubelet\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.048510 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048145    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-slash\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048170    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-modprobe-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-sys\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-host\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-log-socket\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048315    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8j7n\" (UniqueName: \"kubernetes.io/projected/02e9f8c2-3da0-474f-b7db-40bee0e8d743-kube-api-access-d8j7n\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048346    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cnibin\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048370    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fbee35-1719-4228-a543-874c0d3f4b4b-host\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048396    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-systemd-units\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048423    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlhs\" (UniqueName: \"kubernetes.io/projected/85738300-00ba-4ba6-be20-2b259922fd7d-kube-api-access-tqlhs\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048449    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-socket-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048475    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkrm\" (UniqueName: \"kubernetes.io/projected/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kube-api-access-vtkrm\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048503    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-system-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048530    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysconfig\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.048559    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-lib-modules\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.049016    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.049323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.049052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85738300-00ba-4ba6-be20-2b259922fd7d-host-slash\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.050121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.049077    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-sys-fs\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.050121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.049117    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-conf-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.050121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.049143    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-tmp\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.075820 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.075778    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:31:36.076030 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.075976    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:31:36.076030 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076016    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:31:36.076169 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076055    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:31:36.076269 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076243    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:31:36.076356 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076313    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:31:36.076413 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076402    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgbt6\""
Oct 30 12:31:36.076472 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076418    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:31:36.076472 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076383    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:31:36.076472 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076436    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:31:36.076472 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076440    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:31:36.076629 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076611    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:31:36.076664 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076633    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:31:36.076664 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076640    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:31:36.076777 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076761    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:31:36.076824 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076762    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:31:36.076824 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076792    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:31:36.076824 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076822    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:31:36.076924 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076842    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v9g2h\""
Oct 30 12:31:36.076924 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.076842    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:31:36.089948 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.089912    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:31:36.089948 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.089919    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:31:36.090602 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.090567    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fkzfg\""
Oct 30 12:31:36.090602 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.090596    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5fvrw\""
Oct 30 12:31:36.090777 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.090625    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7nlfh\""
Oct 30 12:31:36.090777 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.090667    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kq49t\""
Oct 30 12:31:36.090777 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.090668    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:31:36.090777 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.090680    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:31:36.090777 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.090717    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56d7b\""
Oct 30 12:31:36.150362 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150325    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-script-lib\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.150362 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150374    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.150596 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150469    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.150647 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150615    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnb2q\" (UniqueName: \"kubernetes.io/projected/6662ac83-7fc2-451d-b26b-68d934badad1-kube-api-access-nnb2q\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:36.150698 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150657    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.150698 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150685    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.150772 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150711    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.150772 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150737    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-kubelet-config\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.150869 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150771    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cnibin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.150869 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150819    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-socket-dir-parent\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.150869 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150843    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.151028 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvpv\" (UniqueName: \"kubernetes.io/projected/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-kube-api-access-vnvpv\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.151028 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150917    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-ovn\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151028 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.150988    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdptk\" (UniqueName: \"kubernetes.io/projected/f8924cae-fc43-49bd-9bab-d4bc56220d6a-kube-api-access-sdptk\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151028 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:36.151235 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151026    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-k8s-cni-cncf-io\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.151235 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151048    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-env-overrides\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151235 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151067    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-dbus\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.151235 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.151235 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151096    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-bin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.151235 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151111    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-kubelet\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.151235 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151143    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdjl\" (UniqueName: \"kubernetes.io/projected/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-kube-api-access-2wdjl\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.151235 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151173    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-script-lib\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151302    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-kubelet\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151183    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-kubelet\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151361    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-slash\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151387    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-modprobe-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151412    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-sys\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151429    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-host\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151459    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-log-socket\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8j7n\" (UniqueName: \"kubernetes.io/projected/02e9f8c2-3da0-474f-b7db-40bee0e8d743-kube-api-access-d8j7n\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151506    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cnibin\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151570    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fbee35-1719-4228-a543-874c0d3f4b4b-host\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-systemd-units\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151601    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlhs\" (UniqueName: \"kubernetes.io/projected/85738300-00ba-4ba6-be20-2b259922fd7d-kube-api-access-tqlhs\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151617    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-socket-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkrm\" (UniqueName: \"kubernetes.io/projected/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kube-api-access-vtkrm\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151654    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-system-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.151670 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysconfig\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151692    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-lib-modules\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151737    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151758    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.151828    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-log-socket\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.152036    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cnibin\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.152115    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fbee35-1719-4228-a543-874c0d3f4b4b-host\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.152141    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-systemd-units\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.152251    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-system-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.152494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.152404    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-socket-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.152975 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.152843    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-k8s-cni-cncf-io\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.152975 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.152911    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.153177 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85738300-00ba-4ba6-be20-2b259922fd7d-host-slash\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.153309 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153153    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-dbus\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.153309 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153199    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-bin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.153309 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-sys-fs\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.153309 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153246    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.153309 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153251    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-conf-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.153309 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153288    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-env-overrides\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.153309 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153303    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-tmp\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153318    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-slash\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153337    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-multus\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153351    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysconfig\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.153376    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153427    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd41889-9433-4fdb-8e05-52ea68e32536-agent-certs\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153435    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-modprobe-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153465    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd41889-9433-4fdb-8e05-52ea68e32536-konnectivity-ca\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153504    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.153551 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153536    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-node-log\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153561    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-netd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153601    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovn-node-metrics-cert\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153627    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153654    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153656    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-multus\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153682    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cni-binary-copy\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153694    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-host\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153688    2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.153792    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret podName:f25010f1-45b5-4994-91fa-7de15f04fb08 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:36.65375295 +0000 UTC m=+3.084640925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret") pod "global-pull-secret-syncer-s4gm9" (UID: "f25010f1-45b5-4994-91fa-7de15f04fb08") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153712    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153826    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-lib-modules\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.153839 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153714    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-conf-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.154275 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.153991    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-kubelet\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.154380 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154357    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.154433 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154417    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-sys\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.154476 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154464    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-sys-fs\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.154527 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154497    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85738300-00ba-4ba6-be20-2b259922fd7d-host-slash\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.154527 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154486    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.154607 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154549    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.154607 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154548    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd41889-9433-4fdb-8e05-52ea68e32536-konnectivity-ca\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:36.154607 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154567    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-socket-dir-parent\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.154607 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154579    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-hostroot\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.154747 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154617    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-etc-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.154747 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154634    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cnibin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.154747 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154648    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-tmp-dir\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.154747 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154677    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ng7\" (UniqueName: \"kubernetes.io/projected/46fbee35-1719-4228-a543-874c0d3f4b4b-kube-api-access-k5ng7\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.154747 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154705    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-var-lib-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.154747 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154707    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-node-log\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.154747 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85738300-00ba-4ba6-be20-2b259922fd7d-iptables-alerter-script\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.155019 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:36.155019 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154860    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-conf\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.155019 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154890    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-multus-certs\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.155019 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154919    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-etc-kubernetes\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.155019 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154946    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.155019 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.154978    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-systemd\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.155019 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155006    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-var-lib-kubelet\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155034    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-systemd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155062    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-bin\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-device-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155119    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-netns\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-daemon-config\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155173    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-kubernetes\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155174    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cni-binary-copy\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155220    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155297    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46fbee35-1719-4228-a543-874c0d3f4b4b-serviceca\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-hosts-file\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155409    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-run\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.155480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-tuned\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155510    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9mw\" (UniqueName: \"kubernetes.io/projected/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-kube-api-access-8s9mw\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155540    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-netns\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155569    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-config\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155598    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-registration-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155719    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-registration-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155769    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-daemon-config\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155821    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-kubernetes\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155867    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85738300-00ba-4ba6-be20-2b259922fd7d-iptables-alerter-script\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155950    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-netns\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.155980    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156029    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-kubelet-config\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156106    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156140    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-hostroot\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156161    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.156262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156200    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-etc-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156284    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-etc-kubernetes\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156335    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-var-lib-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156376    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-bin\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156416    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-netd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156453    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-ovn\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156624    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-device-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156711    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-netns\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156719    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156772    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-systemd\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156844    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-var-lib-kubelet\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156758    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-run\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156900    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-systemd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.157048 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.156902    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-hosts-file\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.157209    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.157271    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:36.657254317 +0000 UTC m=+3.088142302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.157452    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46fbee35-1719-4228-a543-874c0d3f4b4b-serviceca\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.157524    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-tmp-dir\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.157552    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-multus-certs\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.157568    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-config\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.157834    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-tmp\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.158095    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd41889-9433-4fdb-8e05-52ea68e32536-agent-certs\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:36.158856 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.158363    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-conf\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.159603 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.159546    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovn-node-metrics-cert\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.160787 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.160699    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-tuned\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.182865 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.182691    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:36.182865 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.182725    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:36.182865 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.182739    2570 projected.go:196] Error preparing data for projected volume kube-api-access-65894 for pod openshift-network-diagnostics/network-check-target-fkhv4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:36.182865 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.182795    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894 podName:061ed4c7-604d-4cea-aedf-f7bbbb7a1c73 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:36.682778425 +0000 UTC m=+3.113666406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-65894" (UniqueName: "kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894") pod "network-check-target-fkhv4" (UID: "061ed4c7-604d-4cea-aedf-f7bbbb7a1c73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:36.184160 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.184128    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8j7n\" (UniqueName: \"kubernetes.io/projected/02e9f8c2-3da0-474f-b7db-40bee0e8d743-kube-api-access-d8j7n\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.186381 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.186288    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnb2q\" (UniqueName: \"kubernetes.io/projected/6662ac83-7fc2-451d-b26b-68d934badad1-kube-api-access-nnb2q\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:36.186762 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.186734    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ng7\" (UniqueName: \"kubernetes.io/projected/46fbee35-1719-4228-a543-874c0d3f4b4b-kube-api-access-k5ng7\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.186875 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.186833    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdjl\" (UniqueName: \"kubernetes.io/projected/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-kube-api-access-2wdjl\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.188977 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.188957    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdptk\" (UniqueName: \"kubernetes.io/projected/f8924cae-fc43-49bd-9bab-d4bc56220d6a-kube-api-access-sdptk\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.189369 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.189347    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkrm\" (UniqueName: \"kubernetes.io/projected/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kube-api-access-vtkrm\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.194295 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.192041    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlhs\" (UniqueName: \"kubernetes.io/projected/85738300-00ba-4ba6-be20-2b259922fd7d-kube-api-access-tqlhs\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.194295 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.193506    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvpv\" (UniqueName: \"kubernetes.io/projected/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-kube-api-access-vnvpv\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.195181 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.195162    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9mw\" (UniqueName: \"kubernetes.io/projected/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-kube-api-access-8s9mw\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.270969 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.270874    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:31:36.273018 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.272989    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:31:36.285146 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:36.285107    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d4d8e7_29c1_425a_87e9_d7fd23dce294.slice/crio-d38ba665120444ee5af3c4a824bf416fab0acaa8aea94b94f66ddbcca61cad40 WatchSource:0}: Error finding container d38ba665120444ee5af3c4a824bf416fab0acaa8aea94b94f66ddbcca61cad40: Status 404 returned error can't find the container with id d38ba665120444ee5af3c4a824bf416fab0acaa8aea94b94f66ddbcca61cad40
Oct 30 12:31:36.285818 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:36.285788    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af58857_77e1_4f65_89e0_3f7fa2ea3a52.slice/crio-ace403decb19cf650361dde39ff050ef6e806d6d0ca5056bf6e4027f653595c2 WatchSource:0}: Error finding container ace403decb19cf650361dde39ff050ef6e806d6d0ca5056bf6e4027f653595c2: Status 404 returned error can't find the container with id ace403decb19cf650361dde39ff050ef6e806d6d0ca5056bf6e4027f653595c2
Oct 30 12:31:36.288728 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.288310    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:31:36.293914 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.293890    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:31:36.302638 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:36.302605    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fbee35_1719_4228_a543_874c0d3f4b4b.slice/crio-0463909a99b82c4211681b939877ecba262f152758f903a703ecff08af62b7a7 WatchSource:0}: Error finding container 0463909a99b82c4211681b939877ecba262f152758f903a703ecff08af62b7a7: Status 404 returned error can't find the container with id 0463909a99b82c4211681b939877ecba262f152758f903a703ecff08af62b7a7
Oct 30 12:31:36.306137 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:36.306095    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1dad4b_c7ea_4f2a_9dfc_c5504675e4e4.slice/crio-a1a92d9fa3b304edb379f0e7e53cd25a8ef9e83dbaefff4c3bcf90e34404bc26 WatchSource:0}: Error finding container a1a92d9fa3b304edb379f0e7e53cd25a8ef9e83dbaefff4c3bcf90e34404bc26: Status 404 returned error can't find the container with id a1a92d9fa3b304edb379f0e7e53cd25a8ef9e83dbaefff4c3bcf90e34404bc26
Oct 30 12:31:36.310186 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.310154    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-89m26"
Oct 30 12:31:36.332811 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.332784    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:36.340590 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.340560    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:36.345555 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:36.345501    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd41889_9433_4fdb_8e05_52ea68e32536.slice/crio-8c20bcc2117a1374ba4c1076841bfd38b29546d7898f5c7dfeb0401c850c8536 WatchSource:0}: Error finding container 8c20bcc2117a1374ba4c1076841bfd38b29546d7898f5c7dfeb0401c850c8536: Status 404 returned error can't find the container with id 8c20bcc2117a1374ba4c1076841bfd38b29546d7898f5c7dfeb0401c850c8536
Oct 30 12:31:36.350262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.350064    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:31:36.352287 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:36.351964    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8924cae_fc43_49bd_9bab_d4bc56220d6a.slice/crio-ad9cccb312fd7a735d377a7a0e2da0bc8bf993c3e5478c8e552116d748824203 WatchSource:0}: Error finding container ad9cccb312fd7a735d377a7a0e2da0bc8bf993c3e5478c8e552116d748824203: Status 404 returned error can't find the container with id ad9cccb312fd7a735d377a7a0e2da0bc8bf993c3e5478c8e552116d748824203
Oct 30 12:31:36.360015 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.359984    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:31:36.364365 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:36.364151    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85738300_00ba_4ba6_be20_2b259922fd7d.slice/crio-bc535eb9a5a55a3eba7ccdb414a6138ba0042246405d8cc38b264f3151e0ff33 WatchSource:0}: Error finding container bc535eb9a5a55a3eba7ccdb414a6138ba0042246405d8cc38b264f3151e0ff33: Status 404 returned error can't find the container with id bc535eb9a5a55a3eba7ccdb414a6138ba0042246405d8cc38b264f3151e0ff33
Oct 30 12:31:36.377627 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:31:36.377587    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f78bb5_5e15_4574_93d6_9cdbdc2c95ba.slice/crio-cdefe5619dd545d6d025a8459ccb04075fa6cee1ec873c1af5154d6a937f6c61 WatchSource:0}: Error finding container cdefe5619dd545d6d025a8459ccb04075fa6cee1ec873c1af5154d6a937f6c61: Status 404 returned error can't find the container with id cdefe5619dd545d6d025a8459ccb04075fa6cee1ec873c1af5154d6a937f6c61
Oct 30 12:31:36.660110 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.660076    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:36.660268 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.660155    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:36.660268 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.660260    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:36.660370 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.660333    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret podName:f25010f1-45b5-4994-91fa-7de15f04fb08 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:37.660315693 +0000 UTC m=+4.091203675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret") pod "global-pull-secret-syncer-s4gm9" (UID: "f25010f1-45b5-4994-91fa-7de15f04fb08") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:36.660464 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.660440    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:36.660554 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.660535    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:37.660514544 +0000 UTC m=+4.091402539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:36.761618 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:36.761306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:36.762015 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.761992    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:36.762087 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.762032    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:36.762087 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.762048    2570 projected.go:196] Error preparing data for projected volume kube-api-access-65894 for pod openshift-network-diagnostics/network-check-target-fkhv4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:36.762192 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:36.762117    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894 podName:061ed4c7-604d-4cea-aedf-f7bbbb7a1c73 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:37.76209544 +0000 UTC m=+4.192983423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-65894" (UniqueName: "kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894") pod "network-check-target-fkhv4" (UID: "061ed4c7-604d-4cea-aedf-f7bbbb7a1c73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:37.005876 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.005720    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:36 +0000 UTC" deadline="2028-05-16 09:47:37.237203288 +0000 UTC"
Oct 30 12:31:37.005876 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.005793    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13533h16m0.231414958s"
Oct 30 12:31:37.193239 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.192612    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:37.193239 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.192770    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:37.225356 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.225176    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerStarted","Data":"cdefe5619dd545d6d025a8459ccb04075fa6cee1ec873c1af5154d6a937f6c61"}
Oct 30 12:31:37.242166 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.242118    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4mvs6" event={"ID":"85738300-00ba-4ba6-be20-2b259922fd7d","Type":"ContainerStarted","Data":"bc535eb9a5a55a3eba7ccdb414a6138ba0042246405d8cc38b264f3151e0ff33"}
Oct 30 12:31:37.255024 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.254873    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"ad9cccb312fd7a735d377a7a0e2da0bc8bf993c3e5478c8e552116d748824203"}
Oct 30 12:31:37.264888 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.264823    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tnrf7" event={"ID":"3dd41889-9433-4fdb-8e05-52ea68e32536","Type":"ContainerStarted","Data":"8c20bcc2117a1374ba4c1076841bfd38b29546d7898f5c7dfeb0401c850c8536"}
Oct 30 12:31:37.273157 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.273114    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-98w6d" event={"ID":"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4","Type":"ContainerStarted","Data":"a1a92d9fa3b304edb379f0e7e53cd25a8ef9e83dbaefff4c3bcf90e34404bc26"}
Oct 30 12:31:37.281778 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.281732    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wjpmw" event={"ID":"46fbee35-1719-4228-a543-874c0d3f4b4b","Type":"ContainerStarted","Data":"0463909a99b82c4211681b939877ecba262f152758f903a703ecff08af62b7a7"}
Oct 30 12:31:37.298891 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.298855    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sq7sj" event={"ID":"6af58857-77e1-4f65-89e0-3f7fa2ea3a52","Type":"ContainerStarted","Data":"ace403decb19cf650361dde39ff050ef6e806d6d0ca5056bf6e4027f653595c2"}
Oct 30 12:31:37.317806 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.316213    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal" event={"ID":"fdd80fd13fc4b06ccce2af5848f9d942","Type":"ContainerStarted","Data":"3661a566248c266c90bfbc677c65b754a7bfa085b8c06021d656d97528e6a781"}
Oct 30 12:31:37.326771 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.326722    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-89m26" event={"ID":"02e9f8c2-3da0-474f-b7db-40bee0e8d743","Type":"ContainerStarted","Data":"1c0c8c44da0c55454e6d8216104ddf1cf3a38822d1093e1449c0b94de4ab6b9c"}
Oct 30 12:31:37.330561 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.330520    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp" event={"ID":"37d4d8e7-29c1-425a-87e9-d7fd23dce294","Type":"ContainerStarted","Data":"d38ba665120444ee5af3c4a824bf416fab0acaa8aea94b94f66ddbcca61cad40"}
Oct 30 12:31:37.672262 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.672168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:37.672470 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.672268    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:37.672470 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.672368    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:37.672470 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.672440    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:39.672418944 +0000 UTC m=+6.103306929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:37.672470 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.672368    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:37.672730 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.672493    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret podName:f25010f1-45b5-4994-91fa-7de15f04fb08 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:39.672480322 +0000 UTC m=+6.103368304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret") pod "global-pull-secret-syncer-s4gm9" (UID: "f25010f1-45b5-4994-91fa-7de15f04fb08") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:37.773984 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:37.773180    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:37.773984 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.773412    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:37.773984 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.773434    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:37.773984 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.773449    2570 projected.go:196] Error preparing data for projected volume kube-api-access-65894 for pod openshift-network-diagnostics/network-check-target-fkhv4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:37.773984 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:37.773514    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894 podName:061ed4c7-604d-4cea-aedf-f7bbbb7a1c73 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:39.773491725 +0000 UTC m=+6.204379710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-65894" (UniqueName: "kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894") pod "network-check-target-fkhv4" (UID: "061ed4c7-604d-4cea-aedf-f7bbbb7a1c73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:38.006476 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:38.006423    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:36 +0000 UTC" deadline="2028-06-21 03:41:17.228265377 +0000 UTC"
Oct 30 12:31:38.006476 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:38.006475    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14391h9m39.221797013s"
Oct 30 12:31:38.194191 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:38.193455    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:38.194191 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:38.193638    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:38.194191 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:38.193686    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:38.194191 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:38.193799    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:38.356009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:38.355913    2570 generic.go:356] "Generic (PLEG): container finished" podID="7813451d010c378fca54e5b82e788cc7" containerID="bb8fe266f5f25173304cfb5935a276af4203d6fcad9ffa6207ccdbd11d05c962" exitCode=0
Oct 30 12:31:38.356999 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:38.356967    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal" event={"ID":"7813451d010c378fca54e5b82e788cc7","Type":"ContainerDied","Data":"bb8fe266f5f25173304cfb5935a276af4203d6fcad9ffa6207ccdbd11d05c962"}
Oct 30 12:31:38.404234 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:38.404163    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal" podStartSLOduration=3.404139297 podStartE2EDuration="3.404139297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:37.361652867 +0000 UTC m=+3.792540858" watchObservedRunningTime="2026-10-30 12:31:38.404139297 +0000 UTC m=+4.835027290"
Oct 30 12:31:39.193138 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:39.192588    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:39.193138 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.192751    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:39.380689 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:39.380229    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal" event={"ID":"7813451d010c378fca54e5b82e788cc7","Type":"ContainerStarted","Data":"e829ade0b8d4f96be97b96d49c1d2646250846470b31f1cace6f59950288d1b4"}
Oct 30 12:31:39.389487 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:39.389435    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerStarted","Data":"f1508d34142b4a5603b32b472a98c59141845677b94bd28aec017c7149ac913f"}
Oct 30 12:31:39.473189 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:39.472783    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal" podStartSLOduration=4.472758783 podStartE2EDuration="4.472758783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:39.426351126 +0000 UTC m=+5.857239117" watchObservedRunningTime="2026-10-30 12:31:39.472758783 +0000 UTC m=+5.903646777"
Oct 30 12:31:39.694311 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:39.693362    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:39.694311 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:39.693447    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:39.694311 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.693585    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:39.694311 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.693655    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret podName:f25010f1-45b5-4994-91fa-7de15f04fb08 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:43.693636328 +0000 UTC m=+10.124524316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret") pod "global-pull-secret-syncer-s4gm9" (UID: "f25010f1-45b5-4994-91fa-7de15f04fb08") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:39.694311 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.694134    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:39.694311 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.694191    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:43.694172865 +0000 UTC m=+10.125060846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:39.794927 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:39.794255    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:39.794927 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.794449    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:39.794927 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.794473    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:39.794927 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.794487    2570 projected.go:196] Error preparing data for projected volume kube-api-access-65894 for pod openshift-network-diagnostics/network-check-target-fkhv4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:39.794927 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:39.794549    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894 podName:061ed4c7-604d-4cea-aedf-f7bbbb7a1c73 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:43.794530237 +0000 UTC m=+10.225418219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-65894" (UniqueName: "kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894") pod "network-check-target-fkhv4" (UID: "061ed4c7-604d-4cea-aedf-f7bbbb7a1c73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:40.199123 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:40.199085    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:40.199659 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:40.199221    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:40.199717 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:40.199675    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:40.199880 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:40.199775    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:40.400176 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:40.398413    2570 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="f1508d34142b4a5603b32b472a98c59141845677b94bd28aec017c7149ac913f" exitCode=0
Oct 30 12:31:40.400176 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:40.399933    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerDied","Data":"f1508d34142b4a5603b32b472a98c59141845677b94bd28aec017c7149ac913f"}
Oct 30 12:31:41.191666 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:41.191632    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:41.191889 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:41.191781    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:41.407945 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:41.407676    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tnrf7" event={"ID":"3dd41889-9433-4fdb-8e05-52ea68e32536","Type":"ContainerStarted","Data":"653a20213edc8e3540a02714888cbe19c53628363a04eeec36def8f505445ee1"}
Oct 30 12:31:42.194944 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:42.194906    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:42.195135 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:42.195061    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:42.195622 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:42.195599    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:42.195756 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:42.195731    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:42.418200 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:42.418149    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wjpmw" event={"ID":"46fbee35-1719-4228-a543-874c0d3f4b4b","Type":"ContainerStarted","Data":"435804a61fdfdadcf7f2aa841cf0e2a3439617d6b1c3f501cca12376e8043b59"}
Oct 30 12:31:42.459146 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:42.459091    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tnrf7" podStartSLOduration=3.385925373 podStartE2EDuration="7.459076649s" totalImagesPullingTime="4.073151276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:41.452432658 +0000 UTC m=+7.883320650" watchObservedRunningTime="2026-10-30 12:31:42.459076649 +0000 UTC m=+8.889964639"
Oct 30 12:31:43.192526 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.192488    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:43.192737 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.192620    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:43.212977 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.212941    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:43.213700 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.213675    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:43.256829 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.256768    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wjpmw" podStartSLOduration=3.23257579 podStartE2EDuration="8.256745386s" totalImagesPullingTime="5.024169596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:42.459803014 +0000 UTC m=+8.890691004" watchObservedRunningTime="2026-10-30 12:31:43.256745386 +0000 UTC m=+9.687633379"
Oct 30 12:31:43.424780 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.424737    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp" event={"ID":"37d4d8e7-29c1-425a-87e9-d7fd23dce294","Type":"ContainerStarted","Data":"cf780624776bf5a1a3124f1334e5b7e996b425cb469b86d8d9032699c59a3dde"}
Oct 30 12:31:43.425299 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.424820    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:43.425838 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.425814    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:31:43.737638 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.737595    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:43.737864 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.737672    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:43.737864 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.737828    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:43.737969 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.737900    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret podName:f25010f1-45b5-4994-91fa-7de15f04fb08 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:51.737878333 +0000 UTC m=+18.168766315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret") pod "global-pull-secret-syncer-s4gm9" (UID: "f25010f1-45b5-4994-91fa-7de15f04fb08") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:43.738407 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.738382    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:43.738522 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.738442    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:51.738426332 +0000 UTC m=+18.169314315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:43.838590 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:43.838537    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:43.838831 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.838802    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:43.838916 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.838837    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:43.838916 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.838855    2570 projected.go:196] Error preparing data for projected volume kube-api-access-65894 for pod openshift-network-diagnostics/network-check-target-fkhv4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:43.839030 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:43.838927    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894 podName:061ed4c7-604d-4cea-aedf-f7bbbb7a1c73 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:51.838903763 +0000 UTC m=+18.269791744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-65894" (UniqueName: "kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894") pod "network-check-target-fkhv4" (UID: "061ed4c7-604d-4cea-aedf-f7bbbb7a1c73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:44.069301 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:44.068780    2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:31:44.132091 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:44.131977    2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:31:44.068800215Z","UUID":"9e069572-7433-45f3-b6f7-7f1ac461d10c","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:31:44.138725 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:44.138684    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:31:44.138725 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:44.138731    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:31:44.194440 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:44.194404    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:44.194611 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:44.194580    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:44.195695 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:44.195673    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:44.195856 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:44.195829    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:44.431049 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:44.430972    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp" event={"ID":"37d4d8e7-29c1-425a-87e9-d7fd23dce294","Type":"ContainerStarted","Data":"ea71612a73884d481e427d656f5baa987a4e4ea37157611517636628235394ff"}
Oct 30 12:31:45.195702 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.195471    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:45.195856 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:45.195797    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:45.435322 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.435281    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-89m26" event={"ID":"02e9f8c2-3da0-474f-b7db-40bee0e8d743","Type":"ContainerStarted","Data":"a4d1ccb775971838af139a94f5d60494abba5f16f52b88c360ef856855c732a8"}
Oct 30 12:31:45.438073 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.438024    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp" event={"ID":"37d4d8e7-29c1-425a-87e9-d7fd23dce294","Type":"ContainerStarted","Data":"a3cbf246e6ab0d232f744a5a484ede2529ddaac8280425b02ed58463ae3a5434"}
Oct 30 12:31:45.440823 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.440792    2570 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="d721300d72a165702116a73239e41434d1600a72a962eed4e1b8721f21580cb6" exitCode=0
Oct 30 12:31:45.440951 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.440873    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerDied","Data":"d721300d72a165702116a73239e41434d1600a72a962eed4e1b8721f21580cb6"}
Oct 30 12:31:45.442924 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.442902    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-98w6d" event={"ID":"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4","Type":"ContainerStarted","Data":"e3fa457c321019991de88d6865877540bf35c53c27f98d27d860c1b68fa7fafb"}
Oct 30 12:31:45.476518 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.476454    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-89m26" podStartSLOduration=2.293967846 podStartE2EDuration="10.476437429s" totalImagesPullingTime="8.182469583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:45.47535715 +0000 UTC m=+11.906245140" watchObservedRunningTime="2026-10-30 12:31:45.476437429 +0000 UTC m=+11.907325417"
Oct 30 12:31:45.555659 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.555545    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-98w6d" podStartSLOduration=2.187285764 podStartE2EDuration="10.555526299s" totalImagesPullingTime="8.368240535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:45.514035489 +0000 UTC m=+11.944923477" watchObservedRunningTime="2026-10-30 12:31:45.555526299 +0000 UTC m=+11.986414284"
Oct 30 12:31:45.597218 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:45.597146    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp" podStartSLOduration=1.966252135 podStartE2EDuration="10.597131066s" totalImagesPullingTime="8.630878931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:45.596523519 +0000 UTC m=+12.027411507" watchObservedRunningTime="2026-10-30 12:31:45.597131066 +0000 UTC m=+12.028019053"
Oct 30 12:31:46.194409 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:46.193507    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:46.194409 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:46.193701    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:46.194409 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:46.193512    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:46.194409 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:46.193847    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:46.448988 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:46.448945    2570 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="cf8d1ba34c7e61259abf65fa23ee245bf3e8113dbe3b9571c4d7e8c109fd5163" exitCode=0
Oct 30 12:31:46.449483 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:46.449062    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerDied","Data":"cf8d1ba34c7e61259abf65fa23ee245bf3e8113dbe3b9571c4d7e8c109fd5163"}
Oct 30 12:31:47.191998 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:47.191952    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:47.192176 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:47.192102    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:47.454043 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:47.454002    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4mvs6" event={"ID":"85738300-00ba-4ba6-be20-2b259922fd7d","Type":"ContainerStarted","Data":"147348d471ca37475d10d733a9d0493830fda7c1f2c88a472e779458046c4602"}
Oct 30 12:31:47.455830 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:47.455790    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sq7sj" event={"ID":"6af58857-77e1-4f65-89e0-3f7fa2ea3a52","Type":"ContainerStarted","Data":"cdb58d09edbd173019c993f24baa5533f234d1bc48455842505824e29fa5474c"}
Oct 30 12:31:47.492786 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:47.492729    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4mvs6" podStartSLOduration=3.811054912 podStartE2EDuration="12.492711088s" totalImagesPullingTime="8.681656176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:47.492462094 +0000 UTC m=+13.923350082" watchObservedRunningTime="2026-10-30 12:31:47.492711088 +0000 UTC m=+13.923599076"
Oct 30 12:31:47.533150 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:47.533089    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sq7sj" podStartSLOduration=1.73486746 podStartE2EDuration="12.533069766s" totalImagesPullingTime="10.798202306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:47.532524034 +0000 UTC m=+13.963412022" watchObservedRunningTime="2026-10-30 12:31:47.533069766 +0000 UTC m=+13.963957754"
Oct 30 12:31:48.192564 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:48.192531    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:48.192741 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:48.192580    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:48.192741 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:48.192715    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:48.192854 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:48.192820    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:48.460973 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:48.460673    2570 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="87304e5d2a4e705f5751d8d6d965a34f6fd35ee6ceff3e0552727ff6f46900c0" exitCode=0
Oct 30 12:31:48.460973 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:48.460753    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerDied","Data":"87304e5d2a4e705f5751d8d6d965a34f6fd35ee6ceff3e0552727ff6f46900c0"}
Oct 30 12:31:49.192372 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:49.192337    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:49.192540 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:49.192489    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:50.192186 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:50.192149    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:50.192581 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:50.192223    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:50.192581 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:50.192324    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:50.192581 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:50.192449    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:50.469143 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:50.469048    2570 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="25f30b5b25b53396cb3bb0f352696f3cc50574abf7c0f8be786ae34bd40c15de" exitCode=0
Oct 30 12:31:50.469143 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:50.469111    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerDied","Data":"25f30b5b25b53396cb3bb0f352696f3cc50574abf7c0f8be786ae34bd40c15de"}
Oct 30 12:31:51.192390 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:51.192351    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:51.192874 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.192466    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:51.473757 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:51.473548    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"b62bd706e08fdc3d393ab1d93f6211bd5554f8e34862a261d380ac279c3fc96f"}
Oct 30 12:31:51.473954 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:51.473853    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"f9efdf3c44f7f77d81870e82a925e8720a754badad95331db0d8a78d71321c39"}
Oct 30 12:31:51.476878 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:51.476851    2570 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="6633d2d70e91016edd5f03b7eb154d83b0e079126763b2fc69a4a341e7a0c885" exitCode=0
Oct 30 12:31:51.476878 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:51.476888    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerDied","Data":"6633d2d70e91016edd5f03b7eb154d83b0e079126763b2fc69a4a341e7a0c885"}
Oct 30 12:31:51.803676 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:51.803639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:51.803866 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:51.803697    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:51.803866 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.803788    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:51.803866 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.803789    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:51.803866 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.803840    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret podName:f25010f1-45b5-4994-91fa-7de15f04fb08 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:07.803827466 +0000 UTC m=+34.234715443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret") pod "global-pull-secret-syncer-s4gm9" (UID: "f25010f1-45b5-4994-91fa-7de15f04fb08") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:51.803866 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.803854    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:07.803847851 +0000 UTC m=+34.234735816 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:51.904367 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:51.904321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:51.904515 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.904484    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:51.904515 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.904502    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:51.904515 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.904512    2570 projected.go:196] Error preparing data for projected volume kube-api-access-65894 for pod openshift-network-diagnostics/network-check-target-fkhv4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:51.904626 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:51.904563    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894 podName:061ed4c7-604d-4cea-aedf-f7bbbb7a1c73 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:07.904546868 +0000 UTC m=+34.335434851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-65894" (UniqueName: "kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894") pod "network-check-target-fkhv4" (UID: "061ed4c7-604d-4cea-aedf-f7bbbb7a1c73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:52.191954 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:52.191862    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:52.192166 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:52.191974    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:52.192166 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:52.192027    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:52.192166 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:52.192127    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:52.481867 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:52.481832    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" event={"ID":"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba","Type":"ContainerStarted","Data":"250554b16659bcd359bdad17282408b09b2f26f03a89f32df6f2db9badcd4450"}
Oct 30 12:31:52.484654 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:52.484628    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"b5f0b3b2fcfa7977259bde83dbae54127df324b111198255e74d9f09385cba97"}
Oct 30 12:31:52.484782 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:52.484696    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"14f00f378ec744c05e0744a909651d3a528dd7914bb67306de1cb0c2898e4e76"}
Oct 30 12:31:52.484782 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:52.484725    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"d9a5ae07e1ca3ebfbf498696794d31e57fbbd360317ac3a5217e31d3c082d32b"}
Oct 30 12:31:52.484782 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:52.484756    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"e93ed264c9f11eaa570c6d1c2870552813fa0666cfb24068a29888a54270dc43"}
Oct 30 12:31:53.192348 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:53.192304    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:53.192525 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:53.192427    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:54.192592 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:54.192552    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:54.193134 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:54.192623    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:54.193134 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:54.192643    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:54.193134 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:54.192717    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:54.492855 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:54.492818    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"7fd57ceb22e28072b81fd87c4a3c307d7fa9162d8649a6e9c7e361718ff62db7"}
Oct 30 12:31:55.191872 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:55.191839    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:55.192088 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:55.191976    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:56.192433 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:56.192307    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:56.192433 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:56.192334    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:56.192907 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:56.192431    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:56.192907 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:56.192482    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:56.501177 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:56.501141    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" event={"ID":"f8924cae-fc43-49bd-9bab-d4bc56220d6a","Type":"ContainerStarted","Data":"99828f1ebd6f215d3aa70262c628fcda63ad62fd3a5da05432b9fc035097ec16"}
Oct 30 12:31:56.501636 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:56.501610    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:56.501767 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:56.501698    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:56.518242 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:56.518197    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:56.589440 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:56.589383    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw" podStartSLOduration=6.638274877 podStartE2EDuration="21.589369804s" totalImagesPullingTime="14.951094927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:56.589272199 +0000 UTC m=+23.020160187" watchObservedRunningTime="2026-10-30 12:31:56.589369804 +0000 UTC m=+23.020257790"
Oct 30 12:31:56.589618 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:56.589574    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n6hfj" podStartSLOduration=11.12513341 podStartE2EDuration="21.589568689s" totalImagesPullingTime="10.464435279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:52.528569874 +0000 UTC m=+18.959457864" watchObservedRunningTime="2026-10-30 12:31:56.589568689 +0000 UTC m=+23.020456675"
Oct 30 12:31:57.192409 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.192366    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:57.192586 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:57.192491    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:57.503455 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.503425    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:57.519402 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.519373    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:31:57.919074 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.918945    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fkhv4"]
Oct 30 12:31:57.919273 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.919133    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:31:57.919411 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:57.919290    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:31:57.922588 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.922545    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7dxkx"]
Oct 30 12:31:57.922724 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.922698    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:57.922838 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:57.922815    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:57.923154 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.923138    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s4gm9"]
Oct 30 12:31:57.923244 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:57.923233    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:57.923312 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:57.923298    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:31:59.192281 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:59.192243    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:31:59.192674 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:31:59.192251    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:31:59.192674 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:59.192366    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:31:59.192674 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:31:59.192444    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:32:00.191936 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:00.191841    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:00.192115 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:00.191977    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:32:01.191631 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:01.191594    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:01.192116 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:01.191594    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:01.192116 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:01.191712    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:32:01.192116 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:01.191771    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:32:02.191804 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:02.191716    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:02.192172 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:02.191838    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:32:03.192596 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:03.192548    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:03.193057 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:03.192552    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:03.193057 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:03.192697    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7dxkx" podUID="6662ac83-7fc2-451d-b26b-68d934badad1"
Oct 30 12:32:03.193057 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:03.192731    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4gm9" podUID="f25010f1-45b5-4994-91fa-7de15f04fb08"
Oct 30 12:32:04.193555 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.193228    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:04.193931 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.193625    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkhv4" podUID="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73"
Oct 30 12:32:04.330115 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.330082    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeReady"
Oct 30 12:32:04.330282 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.330257    2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 12:32:04.387669 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.387633    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vwdb5"]
Oct 30 12:32:04.390779 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.390753    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vwdb5"
Oct 30 12:32:04.392286 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.392260    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"]
Oct 30 12:32:04.395022 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.394996    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77bb4759d7-ztvnj"]
Oct 30 12:32:04.395167 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.395141    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.397840 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.397817    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7tvl5"]
Oct 30 12:32:04.397978 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.397940    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.400827 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.400802    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"]
Oct 30 12:32:04.400961 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.400937    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.403598 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.403577    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"]
Oct 30 12:32:04.404442 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.404418    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"
Oct 30 12:32:04.410625 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.409524    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.410625 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.409599    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.412282 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.412245    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6676c6bb89-m66tl"]
Oct 30 12:32:04.412464 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.412444    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.415304 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.415270    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"]
Oct 30 12:32:04.415499 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.415418    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.418558 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.418536    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"]
Oct 30 12:32:04.418726 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.418667    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.421625 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.421598    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-87dsg"]
Oct 30 12:32:04.421776 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.421720    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.424496 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.424469    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 12:32:04.424863 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.424836    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xngpq\""
Oct 30 12:32:04.424965 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.424903    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zxlk2\""
Oct 30 12:32:04.424965 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.424927    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 12:32:04.425381 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.425364    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"]
Oct 30 12:32:04.425550 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.425528    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:04.428405 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.428380    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ccff9c748-92chp"]
Oct 30 12:32:04.428619 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.428602    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.430053 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.430027    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 12:32:04.431717 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.431695    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lcssw\""
Oct 30 12:32:04.431833 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.431784    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rszcr\""
Oct 30 12:32:04.431833 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.431814    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 12:32:04.432328 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.432302    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"]
Oct 30 12:32:04.432508 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.432474    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.436197 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.435389    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-829vl"]
Oct 30 12:32:04.436197 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.435545    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"
Oct 30 12:32:04.438398 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.438379    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pbb9n"]
Oct 30 12:32:04.438509 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.438495    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-829vl"
Oct 30 12:32:04.440612 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.440591    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 12:32:04.441191 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.441168    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qngfl"]
Oct 30 12:32:04.441334 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.441311    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pbb9n"
Oct 30 12:32:04.444450 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.444370    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vwdb5"]
Oct 30 12:32:04.444450 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.444414    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vd2n6"]
Oct 30 12:32:04.444675 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.444570    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.447675 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.447637    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"]
Oct 30 12:32:04.447775 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.447695    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77bb4759d7-ztvnj"]
Oct 30 12:32:04.447775 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.447740    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7tvl5"]
Oct 30 12:32:04.447891 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.447850    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:04.449862 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.449836    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"]
Oct 30 12:32:04.468288 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.467898    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"]
Oct 30 12:32:04.468288 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.467991    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"]
Oct 30 12:32:04.468582 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.468557    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"]
Oct 30 12:32:04.470750 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.470726    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"]
Oct 30 12:32:04.479939 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.479870    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pbb9n"]
Oct 30 12:32:04.480968 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.480947    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6676c6bb89-m66tl"]
Oct 30 12:32:04.481980 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.481721    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-829vl"]
Oct 30 12:32:04.482638 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.482620    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ccff9c748-92chp"]
Oct 30 12:32:04.483453 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.483436    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"]
Oct 30 12:32:04.484071 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.484057    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vd2n6"]
Oct 30 12:32:04.484710 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.484694    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-87dsg"]
Oct 30 12:32:04.489481 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489452    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgh7v\" (UniqueName: \"kubernetes.io/projected/693d053a-5b5c-4be2-9778-a4255c6f3417-kube-api-access-hgh7v\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.489589 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489495    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.489589 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489570    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.489649 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489595    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztc6\" (UniqueName: \"kubernetes.io/projected/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-kube-api-access-zztc6\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.489649 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489618    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-tmp\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.489649 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489637    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv9vz\" (UniqueName: \"kubernetes.io/projected/c0e41a51-8711-47dd-a04d-af098a4d7fcb-kube-api-access-vv9vz\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.489791 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489702    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7sd\" (UniqueName: \"kubernetes.io/projected/b02a9da2-ad91-4bac-8177-c851dc4a37d3-kube-api-access-rk7sd\") pod \"volume-data-source-validator-58c9f5f46d-4t2tb\" (UID: \"b02a9da2-ad91-4bac-8177-c851dc4a37d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"
Oct 30 12:32:04.489791 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-trusted-ca\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.489854 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489840    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzq8v\" (UniqueName: \"kubernetes.io/projected/51f4f667-c671-472c-ae68-beb10d17b645-kube-api-access-kzq8v\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.489901 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489873    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.489958 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489942    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.490008 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.489977    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-default-certificate\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.490043 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490021    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-snapshots\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.490043 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.490113 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490055    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-serving-cert\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.490113 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490070    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-config\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.490175 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490131    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.490175 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490159    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.490282 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-stats-auth\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.490316 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490299    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.490357 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490338    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.490409 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490398    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.490441 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490421    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.490476 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490439    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlt79\" (UniqueName: \"kubernetes.io/projected/090d2edb-0fb8-4fa0-8d91-d866c06226b3-kube-api-access-dlt79\") pod \"kas-connection-checker-685895c7c9-vwdb5\" (UID: \"090d2edb-0fb8-4fa0-8d91-d866c06226b3\") " pod="kube-system/kas-connection-checker-685895c7c9-vwdb5"
Oct 30 12:32:04.490476 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e41a51-8711-47dd-a04d-af098a4d7fcb-serving-cert\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.490570 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490543    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.490609 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490589    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.490652 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490635    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr685\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.490704 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490685    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.490740 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlgxt\" (UniqueName: \"kubernetes.io/projected/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-kube-api-access-hlgxt\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.490772 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.490756    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.496263 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.496234    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qngfl"]
Oct 30 12:32:04.509297 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509265    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.509297 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509267    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.509512 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509397    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.509512 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509454    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.509512 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509477    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.509512 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509480    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 12:32:04.509674 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509525    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 12:32:04.509674 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509546    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.509674 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509557    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.509674 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509548    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.509674 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.509559    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 12:32:04.510044 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510018    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 12:32:04.510044 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510023    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.510044 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510025    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.510257 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510051    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.510659 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510557    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.510659 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510570    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:32:04.510659 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510589    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.510659 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510617    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.510659 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510641    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.511055 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510832    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.511055 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510956    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.511055 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.510979    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.511055 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.511012    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 12:32:04.511055 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.511032    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 12:32:04.511055 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.511036    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 12:32:04.511351 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.511090    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:04.511351 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.511171    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:32:04.523929 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.523904    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 12:32:04.524051 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.523932    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 12:32:04.536382 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536351    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 12:32:04.536567 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536543    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 12:32:04.536638 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536620    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 12:32:04.536690 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536639    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 12:32:04.536690 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536627    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:32:04.536690 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536644    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 12:32:04.536826 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536697    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:32:04.536826 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536710    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 12:32:04.536826 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536677    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 12:32:04.536826 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536713    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-g89j4\""
Oct 30 12:32:04.536826 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536729    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 12:32:04.536826 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536735    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 12:32:04.536826 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536754    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pwb6k\""
Oct 30 12:32:04.536826 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536798    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bklll\""
Oct 30 12:32:04.537193 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536852    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtx2d\""
Oct 30 12:32:04.537193 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536857    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 12:32:04.537193 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.536943    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5xhj6\""
Oct 30 12:32:04.537193 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.537023    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5lgc6\""
Oct 30 12:32:04.537193 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.537103    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pmx9z\""
Oct 30 12:32:04.537193 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.537164    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mtq9t\""
Oct 30 12:32:04.537395 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.537239    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tpkdl\""
Oct 30 12:32:04.591134 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.591134 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt79\" (UniqueName: \"kubernetes.io/projected/090d2edb-0fb8-4fa0-8d91-d866c06226b3-kube-api-access-dlt79\") pod \"kas-connection-checker-685895c7c9-vwdb5\" (UID: \"090d2edb-0fb8-4fa0-8d91-d866c06226b3\") " pod="kube-system/kas-connection-checker-685895c7c9-vwdb5"
Oct 30 12:32:04.591428 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591158    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e41a51-8711-47dd-a04d-af098a4d7fcb-serving-cert\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.591428 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591188    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjzd\" (UniqueName: \"kubernetes.io/projected/090c5691-fa68-4427-84ed-e7b4797f77c7-kube-api-access-gpjzd\") pod \"network-check-source-6f78d59f99-fc5ft\" (UID: \"090c5691-fa68-4427-84ed-e7b4797f77c7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"
Oct 30 12:32:04.591428 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591235    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.591428 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591256    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.591428 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.591279    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:04.591428 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591306    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmxt\" (UniqueName: \"kubernetes.io/projected/179c5618-551b-4e6e-b0d3-5fa27708fb1f-kube-api-access-5tmxt\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.591738 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.591436    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:05.091413137 +0000 UTC m=+31.522301196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:04.591738 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591512    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr685\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.591738 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591547    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.591738 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591577    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.591738 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591691    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60e34dd-a30c-4f35-9593-1935139f9a47-cert\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:04.591738 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591736    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlgxt\" (UniqueName: \"kubernetes.io/projected/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-kube-api-access-hlgxt\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.592015 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591783    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a04059dd-fc63-4523-829f-c0ce3c4b8129-serving-cert\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.592015 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.592015 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591851    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdlb7\" (UniqueName: \"kubernetes.io/projected/b60e34dd-a30c-4f35-9593-1935139f9a47-kube-api-access-fdlb7\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:04.592015 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh7v\" (UniqueName: \"kubernetes.io/projected/693d053a-5b5c-4be2-9778-a4255c6f3417-kube-api-access-hgh7v\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.592015 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.591943    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 12:32:04.592195 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.592110    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:05.092087847 +0000 UTC m=+31.522975827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : secret "router-metrics-certs-default" not found
Oct 30 12:32:04.592195 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.591975    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.592195 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.592358 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592193    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztc6\" (UniqueName: \"kubernetes.io/projected/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-kube-api-access-zztc6\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.592358 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-tmp\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.592358 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592265    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.592358 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592293    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9vz\" (UniqueName: \"kubernetes.io/projected/c0e41a51-8711-47dd-a04d-af098a4d7fcb-kube-api-access-vv9vz\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.592358 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592316    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.592358 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7sd\" (UniqueName: \"kubernetes.io/projected/b02a9da2-ad91-4bac-8177-c851dc4a37d3-kube-api-access-rk7sd\") pod \"volume-data-source-validator-58c9f5f46d-4t2tb\" (UID: \"b02a9da2-ad91-4bac-8177-c851dc4a37d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"
Oct 30 12:32:04.592588 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592370    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsspd\" (UniqueName: \"kubernetes.io/projected/a04059dd-fc63-4523-829f-c0ce3c4b8129-kube-api-access-xsspd\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.592588 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592397    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-trusted-ca\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.592588 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592412    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.592588 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.592588 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592474    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzq8v\" (UniqueName: \"kubernetes.io/projected/51f4f667-c671-472c-ae68-beb10d17b645-kube-api-access-kzq8v\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.592785 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.592726    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:04.592785 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.592779    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:05.092763935 +0000 UTC m=+31.523651914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:04.592872 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592822    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.592918 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592887    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.592989 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.592971    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.593032 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593016    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/179c5618-551b-4e6e-b0d3-5fa27708fb1f-config-volume\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.593094 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.593079    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:04.593141 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.593099    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6676c6bb89-m66tl: secret "image-registry-tls" not found
Oct 30 12:32:04.593141 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593113    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:04.593268 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.593192    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls podName:fcbd57e7-45d9-472f-9afe-ded1285276b9 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:05.093176823 +0000 UTC m=+31.524064802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls") pod "image-registry-6676c6bb89-m66tl" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9") : secret "image-registry-tls" not found
Oct 30 12:32:04.593325 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593265    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-tmp\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.593389 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593371    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.593455 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593434    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.593514 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593495    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-trusted-ca\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.593564 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593517    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-default-certificate\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.593655 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593635    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bddd1e71-4116-437a-b7ee-873bc218fa0b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.593714 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593687    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.593768 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593714    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-snapshots\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.593768 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593747    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.593862 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593831    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a04059dd-fc63-4523-829f-c0ce3c4b8129-config\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.593914 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdsj7\" (UniqueName: \"kubernetes.io/projected/c5bb30ba-a903-4c97-80a8-6a1f398b6664-kube-api-access-pdsj7\") pod \"kas-connection-checker-685895c7c9-pbb9n\" (UID: \"c5bb30ba-a903-4c97-80a8-6a1f398b6664\") " pod="kube-system/kas-connection-checker-685895c7c9-pbb9n"
Oct 30 12:32:04.593964 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593948    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-serving-cert\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.594022 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.593973    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-config\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.594022 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594002    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.594117 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594029    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.594117 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.594117 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594087    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkwr\" (UniqueName: \"kubernetes.io/projected/bddd1e71-4116-437a-b7ee-873bc218fa0b-kube-api-access-cpkwr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.594293 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594115    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-stats-auth\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.594293 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594139    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5d6d13bc-3169-4d2c-830c-66837db3faf4-nginx-conf\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:04.594293 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594169    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.594293 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594193    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.594293 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594237    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.594293 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594241    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179c5618-551b-4e6e-b0d3-5fa27708fb1f-tmp-dir\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.594293 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594259    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-snapshots\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.594575 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.594575 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594347    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bddd1e71-4116-437a-b7ee-873bc218fa0b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.594575 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594377    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.594575 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594406    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xknz\" (UniqueName: \"kubernetes.io/projected/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539-kube-api-access-4xknz\") pod \"kas-connection-checker-685895c7c9-829vl\" (UID: \"0b3f7eca-7b44-46c3-b1fd-d9bb4f848539\") " pod="kube-system/kas-connection-checker-685895c7c9-829vl"
Oct 30 12:32:04.594575 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594429    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn87g\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.594901 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.594874    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-config\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.595165 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.595137    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.595427 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.595405    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.596319 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.595508    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:05.095495862 +0000 UTC m=+31.526383846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:04.596319 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.595838    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.596319 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.596075    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e41a51-8711-47dd-a04d-af098a4d7fcb-serving-cert\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.596319 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.596118    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.596319 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.596149    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.596319 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.596307    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-default-certificate\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.596935 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.596916    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-stats-auth\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.597504 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.597487    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-serving-cert\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.620543 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.620468    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlt79\" (UniqueName: \"kubernetes.io/projected/090d2edb-0fb8-4fa0-8d91-d866c06226b3-kube-api-access-dlt79\") pod \"kas-connection-checker-685895c7c9-vwdb5\" (UID: \"090d2edb-0fb8-4fa0-8d91-d866c06226b3\") " pod="kube-system/kas-connection-checker-685895c7c9-vwdb5"
Oct 30 12:32:04.620543 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.620509    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr685\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.621933 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.621903    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztc6\" (UniqueName: \"kubernetes.io/projected/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-kube-api-access-zztc6\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:04.621933 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.621930    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9vz\" (UniqueName: \"kubernetes.io/projected/c0e41a51-8711-47dd-a04d-af098a4d7fcb-kube-api-access-vv9vz\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.622245 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.622228    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh7v\" (UniqueName: \"kubernetes.io/projected/693d053a-5b5c-4be2-9778-a4255c6f3417-kube-api-access-hgh7v\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:04.622623 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.622607    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7sd\" (UniqueName: \"kubernetes.io/projected/b02a9da2-ad91-4bac-8177-c851dc4a37d3-kube-api-access-rk7sd\") pod \"volume-data-source-validator-58c9f5f46d-4t2tb\" (UID: \"b02a9da2-ad91-4bac-8177-c851dc4a37d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"
Oct 30 12:32:04.626069 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.626017    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzq8v\" (UniqueName: \"kubernetes.io/projected/51f4f667-c671-472c-ae68-beb10d17b645-kube-api-access-kzq8v\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:04.627469 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.627446    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlgxt\" (UniqueName: \"kubernetes.io/projected/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-kube-api-access-hlgxt\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.628665 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.628643    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:04.694944 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.694854    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjzd\" (UniqueName: \"kubernetes.io/projected/090c5691-fa68-4427-84ed-e7b4797f77c7-kube-api-access-gpjzd\") pod \"network-check-source-6f78d59f99-fc5ft\" (UID: \"090c5691-fa68-4427-84ed-e7b4797f77c7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"
Oct 30 12:32:04.694944 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.694890    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmxt\" (UniqueName: \"kubernetes.io/projected/179c5618-551b-4e6e-b0d3-5fa27708fb1f-kube-api-access-5tmxt\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.695178 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695015    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.695178 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695046    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60e34dd-a30c-4f35-9593-1935139f9a47-cert\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:04.695178 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695074    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a04059dd-fc63-4523-829f-c0ce3c4b8129-serving-cert\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.695178 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695111    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdlb7\" (UniqueName: \"kubernetes.io/projected/b60e34dd-a30c-4f35-9593-1935139f9a47-kube-api-access-fdlb7\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:04.695178 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.695447 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695305    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.695447 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695348    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsspd\" (UniqueName: \"kubernetes.io/projected/a04059dd-fc63-4523-829f-c0ce3c4b8129-kube-api-access-xsspd\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.695447 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695378    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.695447 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695411    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.695634 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695451    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/179c5618-551b-4e6e-b0d3-5fa27708fb1f-config-volume\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.695634 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695478    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:04.695634 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695512    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.695634 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.695522    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:04.695634 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695551    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bddd1e71-4116-437a-b7ee-873bc218fa0b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.695634 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695590    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a04059dd-fc63-4523-829f-c0ce3c4b8129-config\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.695634 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.695614    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:05.195592195 +0000 UTC m=+31.626480173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdsj7\" (UniqueName: \"kubernetes.io/projected/c5bb30ba-a903-4c97-80a8-6a1f398b6664-kube-api-access-pdsj7\") pod \"kas-connection-checker-685895c7c9-pbb9n\" (UID: \"c5bb30ba-a903-4c97-80a8-6a1f398b6664\") " pod="kube-system/kas-connection-checker-685895c7c9-pbb9n"
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695723    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkwr\" (UniqueName: \"kubernetes.io/projected/bddd1e71-4116-437a-b7ee-873bc218fa0b-kube-api-access-cpkwr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695754    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5d6d13bc-3169-4d2c-830c-66837db3faf4-nginx-conf\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695785    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179c5618-551b-4e6e-b0d3-5fa27708fb1f-tmp-dir\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695822    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695860    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bddd1e71-4116-437a-b7ee-873bc218fa0b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695899    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xknz\" (UniqueName: \"kubernetes.io/projected/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539-kube-api-access-4xknz\") pod \"kas-connection-checker-685895c7c9-829vl\" (UID: \"0b3f7eca-7b44-46c3-b1fd-d9bb4f848539\") " pod="kube-system/kas-connection-checker-685895c7c9-829vl"
Oct 30 12:32:04.695966 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.695925    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn87g\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.696404 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.696162    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a04059dd-fc63-4523-829f-c0ce3c4b8129-config\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.696404 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.696267    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/179c5618-551b-4e6e-b0d3-5fa27708fb1f-config-volume\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.696404 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.696287    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 12:32:04.696404 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.696340    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert podName:5d6d13bc-3169-4d2c-830c-66837db3faf4 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:05.196322912 +0000 UTC m=+31.627210890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-87dsg" (UID: "5d6d13bc-3169-4d2c-830c-66837db3faf4") : secret "networking-console-plugin-cert" not found
Oct 30 12:32:04.696956 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.696633    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.696956 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.696677    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.696956 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.696746    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.696956 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.696776    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:04.696956 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.696793    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccff9c748-92chp: secret "image-registry-tls" not found
Oct 30 12:32:04.696956 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:04.696848    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls podName:67405c0f-c406-4e55-ade6-ba7a26a99a38 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:05.196832483 +0000 UTC m=+31.627720459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls") pod "image-registry-6ccff9c748-92chp" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38") : secret "image-registry-tls" not found
Oct 30 12:32:04.697339 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.697119    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179c5618-551b-4e6e-b0d3-5fa27708fb1f-tmp-dir\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.697475 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.697445    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5d6d13bc-3169-4d2c-830c-66837db3faf4-nginx-conf\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:04.697475 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.697464    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bddd1e71-4116-437a-b7ee-873bc218fa0b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.698081 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.698055    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a04059dd-fc63-4523-829f-c0ce3c4b8129-serving-cert\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.698253 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.698232    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.698297 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.698239    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.698711 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.698690    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60e34dd-a30c-4f35-9593-1935139f9a47-cert\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:04.699272 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.699250    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bddd1e71-4116-437a-b7ee-873bc218fa0b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.705304 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.705279    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vwdb5"
Oct 30 12:32:04.722253 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.722202    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdsj7\" (UniqueName: \"kubernetes.io/projected/c5bb30ba-a903-4c97-80a8-6a1f398b6664-kube-api-access-pdsj7\") pod \"kas-connection-checker-685895c7c9-pbb9n\" (UID: \"c5bb30ba-a903-4c97-80a8-6a1f398b6664\") " pod="kube-system/kas-connection-checker-685895c7c9-pbb9n"
Oct 30 12:32:04.722392 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.722376    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjzd\" (UniqueName: \"kubernetes.io/projected/090c5691-fa68-4427-84ed-e7b4797f77c7-kube-api-access-gpjzd\") pod \"network-check-source-6f78d59f99-fc5ft\" (UID: \"090c5691-fa68-4427-84ed-e7b4797f77c7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"
Oct 30 12:32:04.722506 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.722486    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmxt\" (UniqueName: \"kubernetes.io/projected/179c5618-551b-4e6e-b0d3-5fa27708fb1f-kube-api-access-5tmxt\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:04.722612 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.722594    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsspd\" (UniqueName: \"kubernetes.io/projected/a04059dd-fc63-4523-829f-c0ce3c4b8129-kube-api-access-xsspd\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.723013 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.722989    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.724610 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.724590    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn87g\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:04.724899 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.724879    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdlb7\" (UniqueName: \"kubernetes.io/projected/b60e34dd-a30c-4f35-9593-1935139f9a47-kube-api-access-fdlb7\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:04.725152 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.725136    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xknz\" (UniqueName: \"kubernetes.io/projected/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539-kube-api-access-4xknz\") pod \"kas-connection-checker-685895c7c9-829vl\" (UID: \"0b3f7eca-7b44-46c3-b1fd-d9bb4f848539\") " pod="kube-system/kas-connection-checker-685895c7c9-829vl"
Oct 30 12:32:04.729286 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.729265    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkwr\" (UniqueName: \"kubernetes.io/projected/bddd1e71-4116-437a-b7ee-873bc218fa0b-kube-api-access-cpkwr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.735272 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.735242    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:04.739642 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.739608    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"
Oct 30 12:32:04.760009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.759971    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:04.769919 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.769747    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:04.801985 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.801954    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:04.811251 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.809666    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"
Oct 30 12:32:04.817091 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.817035    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-829vl"
Oct 30 12:32:04.822121 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.822061    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pbb9n"
Oct 30 12:32:04.833993 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:04.833741    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:05.025494 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.025472    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vwdb5"]
Oct 30 12:32:05.028075 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.028047    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090d2edb_0fb8_4fa0_8d91_d866c06226b3.slice/crio-1fc05ca77bc319ed63fba43bf518d231b2de14d638da0b0c3424b238ea98c143 WatchSource:0}: Error finding container 1fc05ca77bc319ed63fba43bf518d231b2de14d638da0b0c3424b238ea98c143: Status 404 returned error can't find the container with id 1fc05ca77bc319ed63fba43bf518d231b2de14d638da0b0c3424b238ea98c143
Oct 30 12:32:05.050902 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.050872    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"]
Oct 30 12:32:05.054483 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.054448    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02a9da2_ad91_4bac_8177_c851dc4a37d3.slice/crio-a1d64156d587787ba72c50535bb598061cdeacbf7c85d39df09c2995abacda57 WatchSource:0}: Error finding container a1d64156d587787ba72c50535bb598061cdeacbf7c85d39df09c2995abacda57: Status 404 returned error can't find the container with id a1d64156d587787ba72c50535bb598061cdeacbf7c85d39df09c2995abacda57
Oct 30 12:32:05.081924 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.081445    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7tvl5"]
Oct 30 12:32:05.083564 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.083531    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929ccbc1_e13f_4dba_b00f_11a9c96d62f8.slice/crio-3d15bf254ba722f58ad3aa21403defba71178ef5ceae362ece101745bfb77ce5 WatchSource:0}: Error finding container 3d15bf254ba722f58ad3aa21403defba71178ef5ceae362ece101745bfb77ce5: Status 404 returned error can't find the container with id 3d15bf254ba722f58ad3aa21403defba71178ef5ceae362ece101745bfb77ce5
Oct 30 12:32:05.100944 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.100892    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:05.101099 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.100958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:05.101099 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.101024    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:05.101099 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101042    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:06.101019326 +0000 UTC m=+32.531907323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:05.101099 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101056    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.101098    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101106    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101113    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:06.101096446 +0000 UTC m=+32.531984411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101148    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:06.101136647 +0000 UTC m=+32.532024648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : secret "router-metrics-certs-default" not found
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101165    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101201    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:06.101189276 +0000 UTC m=+32.532077270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.101290    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101363    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101375    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6676c6bb89-m66tl: secret "image-registry-tls" not found
Oct 30 12:32:05.101403 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.101408    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls podName:fcbd57e7-45d9-472f-9afe-ded1285276b9 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:06.101399328 +0000 UTC m=+32.532287292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls") pod "image-registry-6676c6bb89-m66tl" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9") : secret "image-registry-tls" not found
Oct 30 12:32:05.115993 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.115951    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"]
Oct 30 12:32:05.116105 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.116077    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e41a51_8711_47dd_a04d_af098a4d7fcb.slice/crio-1f06ee81df94126b4b32530df390aabe651063871419e45210003125f00ea1a7 WatchSource:0}: Error finding container 1f06ee81df94126b4b32530df390aabe651063871419e45210003125f00ea1a7: Status 404 returned error can't find the container with id 1f06ee81df94126b4b32530df390aabe651063871419e45210003125f00ea1a7
Oct 30 12:32:05.159259 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.159228    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"]
Oct 30 12:32:05.168329 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.168291    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddd1e71_4116_437a_b7ee_873bc218fa0b.slice/crio-dd10a3ee2948075fd79c2f1fe32b86ca3c07f4dcb7c1f1434932193d68a91416 WatchSource:0}: Error finding container dd10a3ee2948075fd79c2f1fe32b86ca3c07f4dcb7c1f1434932193d68a91416: Status 404 returned error can't find the container with id dd10a3ee2948075fd79c2f1fe32b86ca3c07f4dcb7c1f1434932193d68a91416
Oct 30 12:32:05.168913 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.168894    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"]
Oct 30 12:32:05.171328 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.171294    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04059dd_fc63_4523_829f_c0ce3c4b8129.slice/crio-c2e18d6eb3287c9c812387f838104a490c223a3d411b4783dc04eddc90461540 WatchSource:0}: Error finding container c2e18d6eb3287c9c812387f838104a490c223a3d411b4783dc04eddc90461540: Status 404 returned error can't find the container with id c2e18d6eb3287c9c812387f838104a490c223a3d411b4783dc04eddc90461540
Oct 30 12:32:05.181861 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.181829    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pbb9n"]
Oct 30 12:32:05.186156 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.186113    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bb30ba_a903_4c97_80a8_6a1f398b6664.slice/crio-66a6f187390b48b6d8e8f75b1155f773a9ed7a5f6666ffe3fa0cf9ca887f083f WatchSource:0}: Error finding container 66a6f187390b48b6d8e8f75b1155f773a9ed7a5f6666ffe3fa0cf9ca887f083f: Status 404 returned error can't find the container with id 66a6f187390b48b6d8e8f75b1155f773a9ed7a5f6666ffe3fa0cf9ca887f083f
Oct 30 12:32:05.189164 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.189138    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"]
Oct 30 12:32:05.189607 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.189577    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090c5691_fa68_4427_84ed_e7b4797f77c7.slice/crio-a633ffb31b7155f26758ce92d4c60b8e4ab7819b976eabdabbb98bf2e4c44458 WatchSource:0}: Error finding container a633ffb31b7155f26758ce92d4c60b8e4ab7819b976eabdabbb98bf2e4c44458: Status 404 returned error can't find the container with id a633ffb31b7155f26758ce92d4c60b8e4ab7819b976eabdabbb98bf2e4c44458
Oct 30 12:32:05.192218 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.192186    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:05.192400 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.192379    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:05.202616 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.202591    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:05.202910 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.202635    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:05.202910 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.202681    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:05.202910 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.202820    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:05.202910 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.202832    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccff9c748-92chp: secret "image-registry-tls" not found
Oct 30 12:32:05.202910 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.202872    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls podName:67405c0f-c406-4e55-ade6-ba7a26a99a38 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:06.202859806 +0000 UTC m=+32.633747784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls") pod "image-registry-6ccff9c748-92chp" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38") : secret "image-registry-tls" not found
Oct 30 12:32:05.203092 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.202916    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:05.203092 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.202950    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:06.202929154 +0000 UTC m=+32.633817132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:05.203092 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.202984    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 12:32:05.203092 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:05.203003    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert podName:5d6d13bc-3169-4d2c-830c-66837db3faf4 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:06.202997195 +0000 UTC m=+32.633885172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-87dsg" (UID: "5d6d13bc-3169-4d2c-830c-66837db3faf4") : secret "networking-console-plugin-cert" not found
Oct 30 12:32:05.215457 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.215424    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:32:05.215943 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.215918    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sr4bw\""
Oct 30 12:32:05.216095 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.215937    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:32:05.216095 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.216011    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zprfc\""
Oct 30 12:32:05.236323 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.236234    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vd2n6"]
Oct 30 12:32:05.240429 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:05.240395    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60e34dd_a30c_4f35_9593_1935139f9a47.slice/crio-d03b27fdc9d8aad639bff0147a35d56b0f48865ed0901c0be9800cd94693aac7 WatchSource:0}: Error finding container d03b27fdc9d8aad639bff0147a35d56b0f48865ed0901c0be9800cd94693aac7: Status 404 returned error can't find the container with id d03b27fdc9d8aad639bff0147a35d56b0f48865ed0901c0be9800cd94693aac7
Oct 30 12:32:05.241805 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.241775    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-829vl"]
Oct 30 12:32:05.524255 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.524084    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs" event={"ID":"bddd1e71-4116-437a-b7ee-873bc218fa0b","Type":"ContainerStarted","Data":"dd10a3ee2948075fd79c2f1fe32b86ca3c07f4dcb7c1f1434932193d68a91416"}
Oct 30 12:32:05.525414 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.525356    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vd2n6" event={"ID":"b60e34dd-a30c-4f35-9593-1935139f9a47","Type":"ContainerStarted","Data":"d03b27fdc9d8aad639bff0147a35d56b0f48865ed0901c0be9800cd94693aac7"}
Oct 30 12:32:05.527064 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.527041    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pbb9n" event={"ID":"c5bb30ba-a903-4c97-80a8-6a1f398b6664","Type":"ContainerStarted","Data":"f75e000980734d4d5a2fddab0e90a4a444d6ed8fdb613d274cf2774d35fd6e08"}
Oct 30 12:32:05.527180 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.527085    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pbb9n" event={"ID":"c5bb30ba-a903-4c97-80a8-6a1f398b6664","Type":"ContainerStarted","Data":"66a6f187390b48b6d8e8f75b1155f773a9ed7a5f6666ffe3fa0cf9ca887f083f"}
Oct 30 12:32:05.528396 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.528359    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft" event={"ID":"090c5691-fa68-4427-84ed-e7b4797f77c7","Type":"ContainerStarted","Data":"a633ffb31b7155f26758ce92d4c60b8e4ab7819b976eabdabbb98bf2e4c44458"}
Oct 30 12:32:05.529704 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.529677    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb" event={"ID":"b02a9da2-ad91-4bac-8177-c851dc4a37d3","Type":"ContainerStarted","Data":"a1d64156d587787ba72c50535bb598061cdeacbf7c85d39df09c2995abacda57"}
Oct 30 12:32:05.530923 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.530890    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7tvl5" event={"ID":"929ccbc1-e13f-4dba-b00f-11a9c96d62f8","Type":"ContainerStarted","Data":"3d15bf254ba722f58ad3aa21403defba71178ef5ceae362ece101745bfb77ce5"}
Oct 30 12:32:05.532538 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.532508    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-829vl" event={"ID":"0b3f7eca-7b44-46c3-b1fd-d9bb4f848539","Type":"ContainerStarted","Data":"e91ac99f3165271fc05ab0a4a4dad22e8ad419ed3b9c00f6b1f25a4a9ac81666"}
Oct 30 12:32:05.532629 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.532552    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-829vl" event={"ID":"0b3f7eca-7b44-46c3-b1fd-d9bb4f848539","Type":"ContainerStarted","Data":"4f81a461d86764aafb4034d3143211aa49bcbdbc2c8a146cc58b4e8cca73658a"}
Oct 30 12:32:05.533774 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.533752    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" event={"ID":"c0e41a51-8711-47dd-a04d-af098a4d7fcb","Type":"ContainerStarted","Data":"1f06ee81df94126b4b32530df390aabe651063871419e45210003125f00ea1a7"}
Oct 30 12:32:05.535421 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.535399    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vwdb5" event={"ID":"090d2edb-0fb8-4fa0-8d91-d866c06226b3","Type":"ContainerStarted","Data":"ab07e4a27a7c231d91cb627947c7b4ef00d2e89e5e50c94c64803d78a5219e62"}
Oct 30 12:32:05.535507 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.535443    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vwdb5" event={"ID":"090d2edb-0fb8-4fa0-8d91-d866c06226b3","Type":"ContainerStarted","Data":"1fc05ca77bc319ed63fba43bf518d231b2de14d638da0b0c3424b238ea98c143"}
Oct 30 12:32:05.536806 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.536785    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k" event={"ID":"a04059dd-fc63-4523-829f-c0ce3c4b8129","Type":"ContainerStarted","Data":"c2e18d6eb3287c9c812387f838104a490c223a3d411b4783dc04eddc90461540"}
Oct 30 12:32:05.570027 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.569969    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pbb9n" podStartSLOduration=165.569951624 podStartE2EDuration="2m45.569951624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:05.568153805 +0000 UTC m=+31.999041793" watchObservedRunningTime="2026-10-30 12:32:05.569951624 +0000 UTC m=+32.000839607"
Oct 30 12:32:05.662392 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.662328    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-829vl" podStartSLOduration=165.662302876 podStartE2EDuration="2m45.662302876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:05.614936354 +0000 UTC m=+32.045824347" watchObservedRunningTime="2026-10-30 12:32:05.662302876 +0000 UTC m=+32.093190866"
Oct 30 12:32:05.662779 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:05.662746    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vwdb5" podStartSLOduration=165.662738902 podStartE2EDuration="2m45.662738902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:05.661253118 +0000 UTC m=+32.092141105" watchObservedRunningTime="2026-10-30 12:32:05.662738902 +0000 UTC m=+32.093626890"
Oct 30 12:32:06.111836 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.111790    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:06.112009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.111849    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:06.112009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.111913    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:06.112009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.111936    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:06.112009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.111972    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:06.112239 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112104    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:06.112239 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112122    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6676c6bb89-m66tl: secret "image-registry-tls" not found
Oct 30 12:32:06.112239 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112191    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls podName:fcbd57e7-45d9-472f-9afe-ded1285276b9 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:08.112167153 +0000 UTC m=+34.543055135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls") pod "image-registry-6676c6bb89-m66tl" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9") : secret "image-registry-tls" not found
Oct 30 12:32:06.112661 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112641    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:08.112624191 +0000 UTC m=+34.543512171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:06.112752 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112728    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:06.113078 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112762    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:08.112751458 +0000 UTC m=+34.543639437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:06.113078 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112812    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 12:32:06.113078 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112840    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:08.11283124 +0000 UTC m=+34.543719218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : secret "router-metrics-certs-default" not found
Oct 30 12:32:06.113078 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112895    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:06.113078 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.112923    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:08.112914028 +0000 UTC m=+34.543802007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:06.194752 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.194713    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:06.211013 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.210979    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wqghl\""
Oct 30 12:32:06.213388 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.213354    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:06.213524 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.213429    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:06.213524 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:06.213506    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:06.213644 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.213518    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:06.213644 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.213603    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:08.213571422 +0000 UTC m=+34.644459404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:06.213753 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.213707    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:06.213753 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.213721    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccff9c748-92chp: secret "image-registry-tls" not found
Oct 30 12:32:06.213847 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.213768    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls podName:67405c0f-c406-4e55-ade6-ba7a26a99a38 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:08.213753691 +0000 UTC m=+34.644641679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls") pod "image-registry-6ccff9c748-92chp" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38") : secret "image-registry-tls" not found
Oct 30 12:32:06.213847 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.213827    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 12:32:06.213941 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:06.213863    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert podName:5d6d13bc-3169-4d2c-830c-66837db3faf4 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:08.213853035 +0000 UTC m=+34.644741021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-87dsg" (UID: "5d6d13bc-3169-4d2c-830c-66837db3faf4") : secret "networking-console-plugin-cert" not found
Oct 30 12:32:07.550646 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:07.550600    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb" event={"ID":"b02a9da2-ad91-4bac-8177-c851dc4a37d3","Type":"ContainerStarted","Data":"b848b084c977263f64a263a816cf5b338d95211dafa7619b3d648fc8cbcd8a46"}
Oct 30 12:32:07.835076 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:07.834974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:07.835076 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:07.835070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:07.835352 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:07.835324    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:07.835496 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:07.835431    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:39.83540883 +0000 UTC m=+66.266296816 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : secret "metrics-daemon-secret" not found
Oct 30 12:32:07.847039 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:07.846985    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:07.920301 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:07.919944    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:07.936885 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:07.935974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:07.950386 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:07.950253    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:08.022705 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.022504    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.139285    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.139357    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.139430    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.139467    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.139442675 +0000 UTC m=+38.570330657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.139523    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.139536    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.139591    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.139575338 +0000 UTC m=+38.570463320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : secret "router-metrics-certs-default" not found
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.139612    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.139643    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.139681    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.139670259 +0000 UTC m=+38.570558225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.139955    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.140005    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.139990248 +0000 UTC m=+38.570878239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.140071    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.140082    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6676c6bb89-m66tl: secret "image-registry-tls" not found
Oct 30 12:32:08.140237 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.140117    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls podName:fcbd57e7-45d9-472f-9afe-ded1285276b9 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.140106528 +0000 UTC m=+38.570994506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls") pod "image-registry-6676c6bb89-m66tl" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9") : secret "image-registry-tls" not found
Oct 30 12:32:08.240935 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.240888    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:08.241123 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.240960    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:08.241123 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.241022    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:08.241253 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.241235    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:08.241253 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.241253    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccff9c748-92chp: secret "image-registry-tls" not found
Oct 30 12:32:08.241355 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.241313    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls podName:67405c0f-c406-4e55-ade6-ba7a26a99a38 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.241292919 +0000 UTC m=+38.672180884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls") pod "image-registry-6ccff9c748-92chp" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38") : secret "image-registry-tls" not found
Oct 30 12:32:08.241412 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.241348    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:08.241412 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.241382    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 12:32:08.241510 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.241415    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.241393604 +0000 UTC m=+38.672281574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:08.241510 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:08.241435    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert podName:5d6d13bc-3169-4d2c-830c-66837db3faf4 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.241425027 +0000 UTC m=+38.672312993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-87dsg" (UID: "5d6d13bc-3169-4d2c-830c-66837db3faf4") : secret "networking-console-plugin-cert" not found
Oct 30 12:32:08.258508 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.258446    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb" podStartSLOduration=131.461811606 podStartE2EDuration="2m13.258423716s" totalImagesPullingTime="1.79661211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:07.594459908 +0000 UTC m=+34.025347896" watchObservedRunningTime="2026-10-30 12:32:08.258423716 +0000 UTC m=+34.689311705"
Oct 30 12:32:08.259923 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.259899    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s4gm9"]
Oct 30 12:32:08.377005 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.376911    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fkhv4"]
Oct 30 12:32:08.380035 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:08.379948    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061ed4c7_604d_4cea_aedf_f7bbbb7a1c73.slice/crio-e6783470b824e5ada08f8978159f8fa3ab14c870729c3a8ec8b483d07080ac13 WatchSource:0}: Error finding container e6783470b824e5ada08f8978159f8fa3ab14c870729c3a8ec8b483d07080ac13: Status 404 returned error can't find the container with id e6783470b824e5ada08f8978159f8fa3ab14c870729c3a8ec8b483d07080ac13
Oct 30 12:32:08.557441 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.557400    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fkhv4" event={"ID":"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73","Type":"ContainerStarted","Data":"e6783470b824e5ada08f8978159f8fa3ab14c870729c3a8ec8b483d07080ac13"}
Oct 30 12:32:08.559746 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:08.559700    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s4gm9" event={"ID":"f25010f1-45b5-4994-91fa-7de15f04fb08","Type":"ContainerStarted","Data":"a9baf3572559e2aaf610fe99fac3cd83f6924b50d067dcacdbb27c86c5cc0925"}
Oct 30 12:32:10.571240 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.570834    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k" event={"ID":"a04059dd-fc63-4523-829f-c0ce3c4b8129","Type":"ContainerStarted","Data":"5f2890d739f5ee46a85ad9b5640f4255d4b85dd362fbdf83473a1b70735936ba"}
Oct 30 12:32:10.578135 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.577175    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs" event={"ID":"bddd1e71-4116-437a-b7ee-873bc218fa0b","Type":"ContainerStarted","Data":"5505c86ca71a58b5f3db3081fcfa0f13be7102588f17a48478e1a11e111e2a4a"}
Oct 30 12:32:10.581383 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.580014    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7tvl5" event={"ID":"929ccbc1-e13f-4dba-b00f-11a9c96d62f8","Type":"ContainerStarted","Data":"9b57ee02edb0d385d45b0add4474a998e0bcfb33de540bec3d02762d277c34e9"}
Oct 30 12:32:10.584694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.583971    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/0.log"
Oct 30 12:32:10.584694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.584003    2570 generic.go:356] "Generic (PLEG): container finished" podID="c0e41a51-8711-47dd-a04d-af098a4d7fcb" containerID="f949d446ac6f6d270a637bccdd236f439790e6f706c291b5801e2b6c566fcd20" exitCode=255
Oct 30 12:32:10.584694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.584048    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" event={"ID":"c0e41a51-8711-47dd-a04d-af098a4d7fcb","Type":"ContainerDied","Data":"f949d446ac6f6d270a637bccdd236f439790e6f706c291b5801e2b6c566fcd20"}
Oct 30 12:32:10.584694 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.584372    2570 scope.go:122] "RemoveContainer" containerID="f949d446ac6f6d270a637bccdd236f439790e6f706c291b5801e2b6c566fcd20"
Oct 30 12:32:10.636737 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.636353    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k" podStartSLOduration=130.362247753 podStartE2EDuration="2m15.636336944s" totalImagesPullingTime="5.274089191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:10.634885669 +0000 UTC m=+37.065773658" watchObservedRunningTime="2026-10-30 12:32:10.636336944 +0000 UTC m=+37.067224930"
Oct 30 12:32:10.745304 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.744412    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-7tvl5" podStartSLOduration=131.352774713 podStartE2EDuration="2m15.744389828s" totalImagesPullingTime="4.391615115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:10.743158316 +0000 UTC m=+37.174046308" watchObservedRunningTime="2026-10-30 12:32:10.744389828 +0000 UTC m=+37.175277817"
Oct 30 12:32:10.895645 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.895579    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs" podStartSLOduration=130.966493295 podStartE2EDuration="2m15.895555527s" totalImagesPullingTime="4.929062232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:10.790696392 +0000 UTC m=+37.221584378" watchObservedRunningTime="2026-10-30 12:32:10.895555527 +0000 UTC m=+37.326443515"
Oct 30 12:32:10.896751 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.896726    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gl9mb"]
Oct 30 12:32:10.911526 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.911472    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:10.922751 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.922633    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gl9mb"]
Oct 30 12:32:10.936722 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.936530    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rclv\""
Oct 30 12:32:10.936722 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.936620    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:32:10.936722 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.936689    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:32:10.974928 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.974895    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-zks2p-9688t"]
Oct 30 12:32:10.983009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.982772    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:10.983009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.982865    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0654b-10d6-4a98-b6f9-7b679ac028a2-data-volume\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:10.983009 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.982910    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45f0654b-10d6-4a98-b6f9-7b679ac028a2-crio-socket\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:10.983480 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.983455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7rjf\" (UniqueName: \"kubernetes.io/projected/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-api-access-j7rjf\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:10.983581 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.983540    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:10.996188 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:10.995642    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.013028 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.012991    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zks2p-9688t"]
Oct 30 12:32:11.087320 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087278    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0654b-10d6-4a98-b6f9-7b679ac028a2-data-volume\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.087505 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.086896    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0654b-10d6-4a98-b6f9-7b679ac028a2-data-volume\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.087505 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087374    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.087505 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087427    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45f0654b-10d6-4a98-b6f9-7b679ac028a2-crio-socket\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.087505 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087459    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rjf\" (UniqueName: \"kubernetes.io/projected/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-api-access-j7rjf\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.087695 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087527    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45f0654b-10d6-4a98-b6f9-7b679ac028a2-crio-socket\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.087695 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087622    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.087695 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087653    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrfb\" (UniqueName: \"kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.087695 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087685    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.087868 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.087868 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.087810    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.087967 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.087922    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:32:11.088057 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.087994    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:11.088057 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.088035    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:11.588010551 +0000 UTC m=+38.018898532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : configmap "kube-rbac-proxy" not found
Oct 30 12:32:11.088165 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.088081    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:11.588065017 +0000 UTC m=+38.018952985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:11.125422 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.125386    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rjf\" (UniqueName: \"kubernetes.io/projected/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-api-access-j7rjf\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.189319 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.189256    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.189319 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.189309    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrfb\" (UniqueName: \"kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.189711 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.189421    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.189711 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.189481    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.189917 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.189894    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.190662 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.190629    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.191033 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.191007    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.221828 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.221764    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrfb\" (UniqueName: \"kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.300987 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.300908    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"]
Oct 30 12:32:11.314364 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.314327    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"]
Oct 30 12:32:11.314563 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.314489    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"
Oct 30 12:32:11.325069 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.324388    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:11.327382 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.327355    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"
Oct 30 12:32:11.331346 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.331253    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"]
Oct 30 12:32:11.342184 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.342139    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6btbg\""
Oct 30 12:32:11.358278 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.358245    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"]
Oct 30 12:32:11.391284 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.391235    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4n2\" (UniqueName: \"kubernetes.io/projected/fba6806a-680a-4fd0-8d5e-2b02db024040-kube-api-access-kk4n2\") pod \"migrator-66df5d496b-s9s6q\" (UID: \"fba6806a-680a-4fd0-8d5e-2b02db024040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"
Oct 30 12:32:11.391483 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.391459    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhms\" (UniqueName: \"kubernetes.io/projected/b97ac8d4-f132-4fa6-9f7f-d6d050098612-kube-api-access-qqhms\") pod \"migrator-66df5d496b-zl9k6\" (UID: \"b97ac8d4-f132-4fa6-9f7f-d6d050098612\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"
Oct 30 12:32:11.427917 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.427884    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 12:32:11.428198 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.428180    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:11.492155 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.492114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4n2\" (UniqueName: \"kubernetes.io/projected/fba6806a-680a-4fd0-8d5e-2b02db024040-kube-api-access-kk4n2\") pod \"migrator-66df5d496b-s9s6q\" (UID: \"fba6806a-680a-4fd0-8d5e-2b02db024040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"
Oct 30 12:32:11.492356 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.492244    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhms\" (UniqueName: \"kubernetes.io/projected/b97ac8d4-f132-4fa6-9f7f-d6d050098612-kube-api-access-qqhms\") pod \"migrator-66df5d496b-zl9k6\" (UID: \"b97ac8d4-f132-4fa6-9f7f-d6d050098612\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"
Oct 30 12:32:11.520249 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.520198    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4n2\" (UniqueName: \"kubernetes.io/projected/fba6806a-680a-4fd0-8d5e-2b02db024040-kube-api-access-kk4n2\") pod \"migrator-66df5d496b-s9s6q\" (UID: \"fba6806a-680a-4fd0-8d5e-2b02db024040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"
Oct 30 12:32:11.520431 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.520363    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhms\" (UniqueName: \"kubernetes.io/projected/b97ac8d4-f132-4fa6-9f7f-d6d050098612-kube-api-access-qqhms\") pod \"migrator-66df5d496b-zl9k6\" (UID: \"b97ac8d4-f132-4fa6-9f7f-d6d050098612\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"
Oct 30 12:32:11.591346 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.591256    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/1.log"
Oct 30 12:32:11.591860 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.591834    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/0.log"
Oct 30 12:32:11.591990 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.591881    2570 generic.go:356] "Generic (PLEG): container finished" podID="c0e41a51-8711-47dd-a04d-af098a4d7fcb" containerID="5456cdf1ef89b8c177fbf99cbd62f12e9886cfd5ae61fdd92c2309066c57513a" exitCode=255
Oct 30 12:32:11.592049 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.591986    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" event={"ID":"c0e41a51-8711-47dd-a04d-af098a4d7fcb","Type":"ContainerDied","Data":"5456cdf1ef89b8c177fbf99cbd62f12e9886cfd5ae61fdd92c2309066c57513a"}
Oct 30 12:32:11.592102 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.592050    2570 scope.go:122] "RemoveContainer" containerID="f949d446ac6f6d270a637bccdd236f439790e6f706c291b5801e2b6c566fcd20"
Oct 30 12:32:11.593733 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.592327    2570 scope.go:122] "RemoveContainer" containerID="5456cdf1ef89b8c177fbf99cbd62f12e9886cfd5ae61fdd92c2309066c57513a"
Oct 30 12:32:11.593733 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.592571    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:11.593733 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.593411    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.593733 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.593503    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:11.593733 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.593612    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:11.593733 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.593659    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.593646304 +0000 UTC m=+39.024534286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:11.593733 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.593660    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:32:11.593733 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:11.593710    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.593694316 +0000 UTC m=+39.024582283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : configmap "kube-rbac-proxy" not found
Oct 30 12:32:11.652555 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.652305    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"
Oct 30 12:32:11.670304 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:11.670280    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"
Oct 30 12:32:12.062320 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.062008    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zks2p-9688t"]
Oct 30 12:32:12.062578 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:12.062456    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97ac8d4_f132_4fa6_9f7f_d6d050098612.slice/crio-a226ce5143521ddba580e35aba3b0426a15d17a882263f5fb200c43d18d9e5b1 WatchSource:0}: Error finding container a226ce5143521ddba580e35aba3b0426a15d17a882263f5fb200c43d18d9e5b1: Status 404 returned error can't find the container with id a226ce5143521ddba580e35aba3b0426a15d17a882263f5fb200c43d18d9e5b1
Oct 30 12:32:12.062717 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.062694    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"]
Oct 30 12:32:12.065986 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.065958    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"]
Oct 30 12:32:12.067931 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:12.067900    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2ec818_97ad_4634_9c26_d1f6803d439d.slice/crio-720d4f499c17366e77d1fd53d62ad623a985e0656585df587066320bf011490a WatchSource:0}: Error finding container 720d4f499c17366e77d1fd53d62ad623a985e0656585df587066320bf011490a: Status 404 returned error can't find the container with id 720d4f499c17366e77d1fd53d62ad623a985e0656585df587066320bf011490a
Oct 30 12:32:12.077935 ip-10-0-135-24 kubenswrapper[2570]: W1030 12:32:12.077516    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba6806a_680a_4fd0_8d5e_2b02db024040.slice/crio-17765d6d6bea4059edecf8e734b4903cc96402c65c441b1cf06c75418f0c78e5 WatchSource:0}: Error finding container 17765d6d6bea4059edecf8e734b4903cc96402c65c441b1cf06c75418f0c78e5: Status 404 returned error can't find the container with id 17765d6d6bea4059edecf8e734b4903cc96402c65c441b1cf06c75418f0c78e5
Oct 30 12:32:12.208134 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.208036    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:12.208314 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.208237    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:12.208314 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.208261    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6676c6bb89-m66tl: secret "image-registry-tls" not found
Oct 30 12:32:12.208429 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.208336    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls podName:fcbd57e7-45d9-472f-9afe-ded1285276b9 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.208315467 +0000 UTC m=+46.639203450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls") pod "image-registry-6676c6bb89-m66tl" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9") : secret "image-registry-tls" not found
Oct 30 12:32:12.208682 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.208614    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:12.208682 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.208648    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:12.208856 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.208804    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.208783867 +0000 UTC m=+46.639671847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:12.208856 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.208826    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:12.208979 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.208883    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.208868093 +0000 UTC m=+46.639756106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:12.209375 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.209106    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:12.209375 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.209134    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:12.209375 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.209233    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 12:32:12.209375 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.209282    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.209268546 +0000 UTC m=+46.640156528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : secret "router-metrics-certs-default" not found
Oct 30 12:32:12.209375 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.209326    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:12.209682 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.209378    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.209365632 +0000 UTC m=+46.640253620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:12.310415 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.310359    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:12.310584 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.310532    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:12.310584 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.310552    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccff9c748-92chp: secret "image-registry-tls" not found
Oct 30 12:32:12.310689 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.310603    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:12.310689 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.310613    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls podName:67405c0f-c406-4e55-ade6-ba7a26a99a38 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.310594724 +0000 UTC m=+46.741482694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls") pod "image-registry-6ccff9c748-92chp" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38") : secret "image-registry-tls" not found
Oct 30 12:32:12.310689 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.310537    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:12.310689 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.310637    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.310627361 +0000 UTC m=+46.741515340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:12.310689 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.310676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:12.310854 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.310778    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 12:32:12.310854 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.310804    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert podName:5d6d13bc-3169-4d2c-830c-66837db3faf4 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.310797139 +0000 UTC m=+46.741685118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-87dsg" (UID: "5d6d13bc-3169-4d2c-830c-66837db3faf4") : secret "networking-console-plugin-cert" not found
Oct 30 12:32:12.598690 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.598647    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fkhv4" event={"ID":"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73","Type":"ContainerStarted","Data":"e2b0e5713a288fd1a99e674c0eba4509b1e59c9e1716da3d6c80f74347e2ff0e"}
Oct 30 12:32:12.599415 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.599385    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:12.601709 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.601661    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vd2n6" event={"ID":"b60e34dd-a30c-4f35-9593-1935139f9a47","Type":"ContainerStarted","Data":"6047bdd7bb9fa0710f1b3c841d1415b9d76a176c8f457a216b5e7634ae3535df"}
Oct 30 12:32:12.604105 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.603715    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft" event={"ID":"090c5691-fa68-4427-84ed-e7b4797f77c7","Type":"ContainerStarted","Data":"b01a9863637d64aedb9389d1cf2583e8ac5adc21e8ec3b10e3476a259bf0adc2"}
Oct 30 12:32:12.605105 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.605052    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6" event={"ID":"b97ac8d4-f132-4fa6-9f7f-d6d050098612","Type":"ContainerStarted","Data":"a226ce5143521ddba580e35aba3b0426a15d17a882263f5fb200c43d18d9e5b1"}
Oct 30 12:32:12.607455 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.607431    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/1.log"
Oct 30 12:32:12.608548 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.607861    2570 scope.go:122] "RemoveContainer" containerID="5456cdf1ef89b8c177fbf99cbd62f12e9886cfd5ae61fdd92c2309066c57513a"
Oct 30 12:32:12.608548 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.608177    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:12.609722 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.609665    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q" event={"ID":"fba6806a-680a-4fd0-8d5e-2b02db024040","Type":"ContainerStarted","Data":"17765d6d6bea4059edecf8e734b4903cc96402c65c441b1cf06c75418f0c78e5"}
Oct 30 12:32:12.611390 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.611361    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zks2p-9688t" event={"ID":"8c2ec818-97ad-4634-9c26-d1f6803d439d","Type":"ContainerStarted","Data":"fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da"}
Oct 30 12:32:12.611491 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.611417    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zks2p-9688t" event={"ID":"8c2ec818-97ad-4634-9c26-d1f6803d439d","Type":"ContainerStarted","Data":"720d4f499c17366e77d1fd53d62ad623a985e0656585df587066320bf011490a"}
Oct 30 12:32:12.614012 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.613987    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:12.614111 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.614087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:12.614847 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.614823    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:12.614939 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.614900    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:14.614880508 +0000 UTC m=+41.045768479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:12.614939 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.614903    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:32:12.615049 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:12.614946    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:14.614933196 +0000 UTC m=+41.045821161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : configmap "kube-rbac-proxy" not found
Oct 30 12:32:12.651984 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.651491    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fkhv4" podStartSLOduration=33.978203765 podStartE2EDuration="37.651472161s" totalImagesPullingTime="3.673268396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:12.649013054 +0000 UTC m=+39.079901042" watchObservedRunningTime="2026-10-30 12:32:12.651472161 +0000 UTC m=+39.082360149"
Oct 30 12:32:12.807905 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.807389    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft" podStartSLOduration=124.260211067 podStartE2EDuration="2m10.807374348s" totalImagesPullingTime="6.547163281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:12.807016939 +0000 UTC m=+39.237904927" watchObservedRunningTime="2026-10-30 12:32:12.807374348 +0000 UTC m=+39.238262358"
Oct 30 12:32:12.909027 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.908917    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vd2n6" podStartSLOduration=2.414973654 podStartE2EDuration="8.908899885s" totalImagesPullingTime="6.493926231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:12.865541556 +0000 UTC m=+39.296429544" watchObservedRunningTime="2026-10-30 12:32:12.908899885 +0000 UTC m=+39.339787873"
Oct 30 12:32:12.910175 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:12.910130    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-zks2p-9688t" podStartSLOduration=2.9101203939999998 podStartE2EDuration="2.910120394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:12.907783451 +0000 UTC m=+39.338671440" watchObservedRunningTime="2026-10-30 12:32:12.910120394 +0000 UTC m=+39.341008382"
Oct 30 12:32:14.622534 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.622490    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6" event={"ID":"b97ac8d4-f132-4fa6-9f7f-d6d050098612","Type":"ContainerStarted","Data":"32cae07dce0830076e2437bc9b5a9c6e7fefcdae47cbf7dd18fd59bcf97ee408"}
Oct 30 12:32:14.624390 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.624352    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q" event={"ID":"fba6806a-680a-4fd0-8d5e-2b02db024040","Type":"ContainerStarted","Data":"dfbb0ff44f7bb8185ecd9d9a80c9eb18d710ad0beb300832f3af07a776aead9a"}
Oct 30 12:32:14.638182 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.638135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:14.638371 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.638260    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:14.638445 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:14.638426    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:14.638524 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:14.638506    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:18.638488547 +0000 UTC m=+45.069376534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:14.638918 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.638880    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:14.671490 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.671426    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6" podStartSLOduration=1.839767621 podStartE2EDuration="3.671409561s" totalImagesPullingTime="1.83164194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:14.671003302 +0000 UTC m=+41.101891290" watchObservedRunningTime="2026-10-30 12:32:14.671409561 +0000 UTC m=+41.102297548"
Oct 30 12:32:14.713042 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.712980    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q" podStartSLOduration=1.8999237500000001 podStartE2EDuration="3.712964965s" totalImagesPullingTime="1.813041215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:14.711882084 +0000 UTC m=+41.142770070" watchObservedRunningTime="2026-10-30 12:32:14.712964965 +0000 UTC m=+41.143852961"
Oct 30 12:32:14.760530 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.760487    2570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:14.760698 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.760556    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:14.760937 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.760925    2570 scope.go:122] "RemoveContainer" containerID="5456cdf1ef89b8c177fbf99cbd62f12e9886cfd5ae61fdd92c2309066c57513a"
Oct 30 12:32:14.761148 ip-10-0-135-24 kubenswrapper[2570]: E1030 12:32:14.761128    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:14.995684 ip-10-0-135-24 kubenswrapper[2570]: I1030 12:32:14.995655    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-98w6d_ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4/dns-node-resolver/0.log"
Oct 30 12:32:15.445571 ip-10-0-135-24 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 12:32:15.464844 ip-10-0-135-24 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 12:32:15.465121 ip-10-0-135-24 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 12:32:15.465195 ip-10-0-135-24 systemd[1]: kubelet.service: Consumed 3.443s CPU time.
Oct 30 12:32:15.480868 ip-10-0-135-24 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:32:15.598179 ip-10-0-135-24 kubenswrapper[4993]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:32:15.598179 ip-10-0-135-24 kubenswrapper[4993]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:32:15.598179 ip-10-0-135-24 kubenswrapper[4993]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:32:15.598179 ip-10-0-135-24 kubenswrapper[4993]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:32:15.600381 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600272    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:15.600381 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600379    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:15.600381 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600383    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:15.600381 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600387    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600398    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600402    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600405    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600408    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600411    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600414    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600417    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600420    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600423    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600426    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600433    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600437    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600440    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600443    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600446    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600448    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600451    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600454    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:15.600511 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600456    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600459    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600461    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600464    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600467    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600470    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600473    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600476    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600478    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600481    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600485    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600489    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600492    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600495    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600498    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600500    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600503    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600506    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600509    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600511    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:15.600986 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600514    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600517    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600520    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600522    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600525    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600528    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600534    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600541    4993 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600548    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600553    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600558    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600562    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600566    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600570    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600576    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600580    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600583    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600588    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600591    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:15.601496 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600593    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600596    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600599    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600601    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600605    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600608    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600611    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600614    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600616    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600619    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600622    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600625    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600628    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600631    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600634    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600636    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600639    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600642    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600645    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:15.601960 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600648    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600650    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600653    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600656    4993 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600659    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600662    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600665    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600667    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600674    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600677    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600680    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600683    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600686    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600688    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600691    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600694    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600697    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600699    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600703    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600707    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:15.602437 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600710    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600713    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600715    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600718    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600721    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.600723    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601590    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601599    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601603    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601606    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601609    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601612    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601615    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601618    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601621    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601624    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601626    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601629    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601632    4993 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601634    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:15.602950 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601639    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601642    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601646    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601648    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601651    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601654    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601657    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601660    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601663    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601666    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601669    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601672    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601675    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601677    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601680    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601683    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601686    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601690    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601694    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601697    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:15.603469 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601700    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601703    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601705    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601708    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601710    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601713    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601716    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601718    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601721    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601724    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601726    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601729    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601732    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601735    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601738    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601740    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601743    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601746    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601748    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601751    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:15.603976 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601754    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601756    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601759    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601762    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601765    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601768    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601770    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601773    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601775    4993 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601778    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601781    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601783    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601786    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601788    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601791    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601793    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601796    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601804    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601807    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601810    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:15.604480 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601813    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601815    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601818    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601820    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601824    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601828    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601832    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601835    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601838    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601840    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601843    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601845    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601848    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601851    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601854    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601856    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601859    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601862    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601866    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:15.605005 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601868    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601871    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601873    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601876    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601883    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601888    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601891    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601893    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601896    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601899    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601903    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601906    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.601908    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602417    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602423    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602426    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602429    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602432    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602435    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602438    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:15.605490 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602441    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602444    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602447    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602450    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602453    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602456    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602459    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602462    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602465    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602468    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602471    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602474    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602477    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602479    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602482    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602485    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602490    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602493    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602495    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:15.605980 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602498    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602501    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602504    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602507    4993 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602510    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602512    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602515    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602518    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602520    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602523    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602526    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602528    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602531    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602533    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602536    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602539    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602542    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602544    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602547    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602551    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:15.606589 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602554    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602557    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602561    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602565    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602568    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602571    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602574    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602576    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602579    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602583    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602586    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602588    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602591    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602594    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602596    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602599    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602602    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602605    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602607    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:15.607342 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602610    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602612    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602615    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602617    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602620    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602623    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602626    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602629    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602631    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602634    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602636    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602639    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602642    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602644    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602647    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602649    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602652    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602654    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602657    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602660    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:15.608168 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602663    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602665    4993 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602670    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602673    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602675    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602678    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602681    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602684    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602686    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602689    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602692    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602696    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602699    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602702    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602705    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602709    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602712    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602715    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602717    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602720    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:15.608794 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.602722    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602817    4993 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602825    4993 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602831    4993 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602836    4993 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602844    4993 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602849    4993 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602855    4993 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602862    4993 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602867    4993 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602871    4993 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602877    4993 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602881    4993 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602884    4993 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602889    4993 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602892    4993 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602896    4993 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602899    4993 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602902    4993 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602907    4993 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602910    4993 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602913    4993 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602916    4993 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602920    4993 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:32:15.609308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602924    4993 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602927    4993 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602930    4993 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602934    4993 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602937    4993 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602941    4993 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602944    4993 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602948    4993 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602951    4993 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602956    4993 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602959    4993 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602962    4993 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602966    4993 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602971    4993 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602974    4993 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602978    4993 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602982    4993 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602985    4993 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602988    4993 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602992    4993 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.602998    4993 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603001    4993 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603006    4993 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603009    4993 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603013    4993 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:32:15.609898 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603016    4993 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603019    4993 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603022    4993 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603026    4993 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603029    4993 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603032    4993 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603037    4993 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603041    4993 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603044    4993 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603047    4993 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603051    4993 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603054    4993 flags.go:64] FLAG: --help="false"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603058    4993 flags.go:64] FLAG: --hostname-override="ip-10-0-135-24.ec2.internal"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603061    4993 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603064    4993 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603068    4993 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603072    4993 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603076    4993 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603079    4993 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603082    4993 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603086    4993 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603090    4993 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603093    4993 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603097    4993 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:32:15.610563 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603100    4993 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603103    4993 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603106    4993 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603109    4993 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603113    4993 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603118    4993 flags.go:64] FLAG: --lock-file=""
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603121    4993 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603124    4993 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603128    4993 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603134    4993 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603137    4993 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603140    4993 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603143    4993 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603146    4993 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603150    4993 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603153    4993 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603156    4993 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603166    4993 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603169    4993 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603173    4993 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603176    4993 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603180    4993 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603183    4993 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603186    4993 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603189    4993 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:32:15.611140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603193    4993 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603196    4993 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603216    4993 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603221    4993 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603224    4993 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603228    4993 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603231    4993 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603234    4993 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603237    4993 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603241    4993 flags.go:64] FLAG: --port="10250"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603244    4993 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603247    4993 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d8a3246043739615"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603253    4993 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603256    4993 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603259    4993 flags.go:64] FLAG: --register-node="true"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603262    4993 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603266    4993 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603269    4993 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603272    4993 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603275    4993 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603280    4993 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603283    4993 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603286    4993 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603289    4993 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603292    4993 flags.go:64] FLAG: --runonce="false"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603296    4993 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:32:15.611774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603299    4993 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603303    4993 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603306    4993 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603309    4993 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603313    4993 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603316    4993 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603320    4993 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603323    4993 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603326    4993 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603337    4993 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603341    4993 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603344    4993 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603347    4993 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603350    4993 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603356    4993 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603359    4993 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603363    4993 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603368    4993 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603372    4993 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603376    4993 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603379    4993 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603382    4993 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603385    4993 flags.go:64] FLAG: --v="2"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603390    4993 flags.go:64] FLAG: --version="false"
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603395    4993 flags.go:64] FLAG: --vmodule=""
Oct 30 12:32:15.612454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603399    4993 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603403    4993 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603569    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603574    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603577    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603581    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603584    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603587    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603590    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603593    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603595    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603598    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603601    4993 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603604    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603607    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603609    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603614    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603616    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603619    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603622    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:15.613064 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603624    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603627    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603630    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603632    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603635    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603639    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603642    4993 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603646    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603649    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603651    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603654    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603657    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603660    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603663    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603665    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603668    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603671    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603675    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603678    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603681    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:15.613585 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603684    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603687    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603690    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603692    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603695    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603698    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603700    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603703    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603707    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603710    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603713    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603716    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603718    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603721    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603723    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603726    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603729    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603733    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603735    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603738    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:15.614135 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603741    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603744    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603746    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603749    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603752    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603754    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603757    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603760    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603762    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603765    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603768    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603770    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603773    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603775    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603778    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603781    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603783    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603786    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603789    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:15.614741 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603792    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603796    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603798    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603801    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603804    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603807    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603809    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603812    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603814    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603817    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603821    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603824    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603826    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603829    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603832    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603835    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603838    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603841    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603846    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:15.615468 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603849    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603852    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603855    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603858    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603860    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603864    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603866    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603869    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603872    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603874    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.603901    4993 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603941    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603946    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603949    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603952    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:15.616235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603958    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603962    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603965    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603967    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603970    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603973    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603975    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603978    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603983    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603986    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603989    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603991    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603994    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.603997    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604000    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604003    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604006    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604008    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604011    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604013    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:15.616869 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604016    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604019    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604021    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604024    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604027    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604029    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604032    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604035    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604037    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604040    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604043    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604045    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604049    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604052    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604055    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604057    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604060    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604063    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604066    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604068    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:15.617737 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604072    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604074    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604077    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604080    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604082    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604085    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604089    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604092    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604094    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604097    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604100    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604102    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604105    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604108    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604110    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604113    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604115    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604118    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604120    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:15.618768 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604124    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604128    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604133    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604136    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604139    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604144    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604147    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604150    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604152    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604156    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604158    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604161    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604164    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604167    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604170    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604172    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604175    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604178    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604181    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:15.619605 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604183    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604188    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604191    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604193    4993 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604196    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604199    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604201    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604219    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604221    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604224    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604226    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604229    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604232    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604235    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604238    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604240    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604243    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604247    4993 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604251    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604254    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:15.620621 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604257    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604259    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604262    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.604265    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.609956    4993 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.609980    4993 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.610007    4993 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.610012    4993 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610079    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610086    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610090    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610093    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610096    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610098    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610101    4993 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610104    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610107    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610110    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610113    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:15.621517 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610116    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610119    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610122    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610124    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610127    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610130    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610133    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610135    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610138    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610141    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610143    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610146    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610149    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610152    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610155    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610158    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610161    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610164    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610167    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610170    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:15.622339 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610172    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610175    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610178    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610181    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610183    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610186    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610189    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610191    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610194    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610197    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610199    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610216    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610220    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610223    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610225    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610231    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610234    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610237    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610241    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:15.623157 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610246    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610249    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610252    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610255    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610258    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610261    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610264    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610266    4993 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610270    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610272    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610275    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610278    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610280    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610283    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610286    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610289    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610292    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610294    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610298    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610300    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:15.623999 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610303    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610305    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610308    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610311    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610313    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610316    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610319    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610322    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610325    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610329    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610332    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610336    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610341    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610345    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610350    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610354    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610358    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610362    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610366    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:15.624871 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610370    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610375    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610379    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610383    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610387    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610391    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610394    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610398    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610401    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610404    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610407    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610410    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610413    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610416    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610419    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610422    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:15.625711 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610425    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.610457    4993 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610615    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610621    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610624    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610627    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610630    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610634    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610637    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610640    4993 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610643    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610646    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610648    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610651    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610654    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610657    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:15.626410 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610660    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610664    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610666    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610669    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610672    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610674    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610677    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610680    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610684    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610688    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610691    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610694    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610697    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610699    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610702    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610705    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610708    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610710    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610713    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:15.627098 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610716    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610719    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610721    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610724    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610727    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610730    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610733    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610736    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610739    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610741    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610744    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610747    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610749    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610752    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610754    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610757    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610760    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610762    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610765    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:15.628015 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610768    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610771    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610773    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610776    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610779    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610782    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610785    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610787    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610790    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610792    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610795    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610798    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610801    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610804    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610806    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610809    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610812    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610816    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610820    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:15.628957 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610827    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610831    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610834    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610837    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610839    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610842    4993 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610845    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610848    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610850    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610853    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610856    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610859    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610861    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610867    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610870    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610873    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610875    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610878    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610881    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610884    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:15.629831 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610886    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610889    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610892    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610895    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610898    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610901    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610904    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610906    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610909    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610912    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610915    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610917    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610920    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610923    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610926    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.610950    4993 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:32:15.630481 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610984    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610988    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610992    4993 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610996    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.610998    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611001    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611004    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611007    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611010    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611014    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611017    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611020    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611022    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611025    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611028    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611031    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611033    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611037    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611041    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:15.630904 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611044    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611048    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611051    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611054    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611056    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611059    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611062    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611065    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611067    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611070    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611073    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611076    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611078    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611081    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611084    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611087    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611089    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611092    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611094    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611097    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:15.631457 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611100    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611102    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611111    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611113    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611116    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611119    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611122    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611125    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611128    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611131    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611134    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611136    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611140    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611143    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611146    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611149    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611152    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611154    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611157    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:15.631946 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611160    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611162    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611165    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611167    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611170    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611173    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611178    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611182    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611184    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611188    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611191    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611194    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611196    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611199    4993 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611215    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611219    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611222    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611225    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611228    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:15.632518 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611230    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611233    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611236    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611239    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611242    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611244    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611247    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611250    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611252    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611255    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611258    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611260    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611263    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611266    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611268    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611271    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611277    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611280    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611282    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611286    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:15.633055 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611289    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611292    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611295    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611298    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611301    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611304    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611307    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611310    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:15.611313    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.611383    4993 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.612482    4993 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.612557    4993 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.612756    4993 server.go:1014] "Starting client certificate rotation"
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.612895    4993 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.612950    4993 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:26:34 +0000 UTC" deadline="2028-04-01 10:12:13.082196064 +0000 UTC"
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.612990    4993 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12453h39m57.469208266s"
Oct 30 12:32:15.633783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.613251    4993 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:32:15.634189 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.614488    4993 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:32:15.634189 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.616072    4993 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:32:15.634189 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.618160    4993 log.go:25] "Validated CRI v1 image API"
Oct 30 12:32:15.634189 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.618970    4993 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:32:15.634189 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.621232    4993 fs.go:135] Filesystem UUIDs: map[6db28a20-2ee0-425e-837f-7ab8bb9d87f3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8cee4342-7991-4ac6-aac6-088d3c347a82:/dev/nvme0n1p3]
Oct 30 12:32:15.634377 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.621257    4993 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0463909a99b82c4211681b939877ecba262f152758f903a703ecff08af62b7a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0463909a99b82c4211681b939877ecba262f152758f903a703ecff08af62b7a7/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17765d6d6bea4059edecf8e734b4903cc96402c65c441b1cf06c75418f0c78e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17765d6d6bea4059edecf8e734b4903cc96402c65c441b1cf06c75418f0c78e5/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c0c8c44da0c55454e6d8216104ddf1cf3a38822d1093e1449c0b94de4ab6b9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c0c8c44da0c55454e6d8216104ddf1cf3a38822d1093e1449c0b94de4ab6b9c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f06ee81df94126b4b32530df390aabe651063871419e45210003125f00ea1a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f06ee81df94126b4b32530df390aabe651063871419e45210003125f00ea1a7/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fc05ca77bc319ed63fba43bf518d231b2de14d638da0b0c3424b238ea98c143/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fc05ca77bc319ed63fba43bf518d231b2de14d638da0b0c3424b238ea98c143/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d15bf254ba722f58ad3aa21403defba71178ef5ceae362ece101745bfb77ce5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d15bf254ba722f58ad3aa21403defba71178ef5ceae362ece101745bfb77ce5/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f81a461d86764aafb4034d3143211aa49bcbdbc2c8a146cc58b4e8cca73658a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f81a461d86764aafb4034d3143211aa49bcbdbc2c8a146cc58b4e8cca73658a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66a6f187390b48b6d8e8f75b1155f773a9ed7a5f6666ffe3fa0cf9ca887f083f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66a6f187390b48b6d8e8f75b1155f773a9ed7a5f6666ffe3fa0cf9ca887f083f/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/720d4f499c17366e77d1fd53d62ad623a985e0656585df587066320bf011490a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/720d4f499c17366e77d1fd53d62ad623a985e0656585df587066320bf011490a/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8670e2183233dad6028f9f37b60801aefef0cf2a99a0f93f76cab709b9537bdf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8670e2183233dad6028f9f37b60801aefef0cf2a99a0f93f76cab709b9537bdf/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c20bcc2117a1374ba4c1076841bfd38b29546d7898f5c7dfeb0401c850c8536/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c20bcc2117a1374ba4c1076841bfd38b29546d7898f5c7dfeb0401c850c8536/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b27de364aa5303a86c83db913102d4c4d79f2b4c36a4f69cf6c8d9e3e22ef31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b27de364aa5303a86c83db913102d4c4d79f2b4c36a4f69cf6c8d9e3e22ef31/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1a92d9fa3b304edb379f0e7e53cd25a8ef9e83dbaefff4c3bcf90e34404bc26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1a92d9fa3b304edb379f0e7e53cd25a8ef9e83dbaefff4c3bcf90e34404bc26/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1d64156d587787ba72c50535bb598061cdeacbf7c85d39df09c2995abacda57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1d64156d587787ba72c50535bb598061cdeacbf7c85d39df09c2995abacda57/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a226ce5143521ddba580e35aba3b0426a15d17a882263f5fb200c43d18d9e5b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a226ce5143521ddba580e35aba3b0426a15d17a882263f5fb200c43d18d9e5b1/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a633ffb31b7155f26758ce92d4c60b8e4ab7819b976eabdabbb98bf2e4c44458/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a633ffb31b7155f26758ce92d4c60b8e4ab7819b976eabdabbb98bf2e4c44458/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9baf3572559e2aaf610fe99fac3cd83f6924b50d067dcacdbb27c86c5cc0925/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9baf3572559e2aaf610fe99fac3cd83f6924b50d067dcacdbb27c86c5cc0925/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad9cccb312fd7a735d377a7a0e2da0bc8bf993c3e5478c8e552116d748824203/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad9cccb312fd7a735d377a7a0e2da0bc8bf993c3e5478c8e552116d748824203/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc535eb9a5a55a3eba7ccdb414a6138ba0042246405d8cc38b264f3151e0ff33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc535eb9a5a55a3eba7ccdb414a6138ba0042246405d8cc38b264f3151e0ff33/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2e18d6eb3287c9c812387f838104a490c223a3d411b4783dc04eddc90461540/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2e18d6eb3287c9c812387f838104a490c223a3d411b4783dc04eddc90461540/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdefe5619dd545d6d025a8459ccb04075fa6cee1ec873c1af5154d6a937f6c61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdefe5619dd545d6d025a8459ccb04075fa6cee1ec873c1af5154d6a937f6c61/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d03b27fdc9d8aad639bff0147a35d56b0f48865ed0901c0be9800cd94693aac7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d03b27fdc9d8aad639bff0147a35d56b0f48865ed0901c0be9800cd94693aac7/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d38ba665120444ee5af3c4a824bf416fab0acaa8aea94b94f66ddbcca61cad40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d38ba665120444ee5af3c4a824bf416fab0acaa8aea94b94f66ddbcca61cad40/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd10a3ee2948075fd79c2f1fe32b86ca3c07f4dcb7c1f1434932193d68a91416/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd10a3ee2948075fd79c2f1fe32b86ca3c07f4dcb7c1f1434932193d68a91416/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6783470b824e5ada08f8978159f8fa3ab14c870729c3a8ec8b483d07080ac13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6783470b824e5ada08f8978159f8fa3ab14c870729c3a8ec8b483d07080ac13/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02e9f8c2-3da0-474f-b7db-40bee0e8d743/volumes/kubernetes.io~projected/kube-api-access-d8j7n:{mountpoint:/var/lib/kubelet/pods/02e9f8c2-3da0-474f-b7db-40bee0e8d743/volumes/kubernetes.io~projected/kube-api-access-d8j7n major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73/volumes/kubernetes.io~projected/kube-api-access-65894:{mountpoint:/var/lib/kubelet/pods/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73/volumes/kubernetes.io~projected/kube-api-access-65894 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/090c5691-fa68-4427-84ed-e7b4797f77c7/volumes/kubernetes.io~projected/kube-api-access-gpjzd:{mountpoint:/var/lib/kubelet/pods/090c5691-fa68-4427-84ed-e7b4797f77c7/volumes/kubernetes.io~projected/kube-api-access-gpjzd major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/090d2edb-0fb8-4fa0-8d91-d866c06226b3/volumes/kubernetes.io~projected/kube-api-access-dlt79:{mountpoint:/var/lib/kubelet/pods/090d2edb-0fb8-4fa0-8d91-d866c06226b3/volumes/kubernetes.io~projected/kube-api-access-dlt79 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539/volumes/kubernetes.io~projected/kube-api-access-4xknz:{mountpoint:/var/lib/kubelet/pods/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539/volumes/kubernetes.io~projected/kube-api-access-4xknz major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/179c5618-551b-4e6e-b0d3-5fa27708fb1f/volumes/kubernetes.io~projected/kube-api-access-5tmxt:{mountpoint:/var/lib/kubelet/pods/179c5618-551b-4e6e-b0d3-5fa27708fb1f/volumes/kubernetes.io~projected/kube-api-access-5tmxt major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37d4d8e7-29c1-425a-87e9-d7fd23dce294/volumes/kubernetes.io~projected/kube-api-access-vtkrm:{mountpoint:/var/lib/kubelet/pods/37d4d8e7-29c1-425a-87e9-d7fd23dce294/volumes/kubernetes.io~projected/kube-api-access-vtkrm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3dd41889-9433-4fdb-8e05-52ea68e32536/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3dd41889-9433-4fdb-8e05-52ea68e32536/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45f0654b-10d6-4a98-b6f9-7b679ac028a2/volumes/kubernetes.io~projected/kube-api-access-j7rjf:{mountpoint:/var/lib/kubelet/pods/45f0654b-10d6-4a98-b6f9-7b679ac028a2/volumes/kubernetes.io~projected/kube-api-access-j7rjf major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46fbee35-1719-4228-a543-874c0d3f4b4b/volumes/kubernetes.io~projected/kube-api-access-k5ng7:{mountpoint:/var/lib/kubelet/pods/46fbee35-1719-4228-a543-874c0d3f4b4b/volumes/kubernetes.io~projected/kube-api-access-k5ng7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~projected/kube-api-access-kzq8v:{mountpoint:/var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~projected/kube-api-access-kzq8v major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~secret/stats-auth major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba/volumes/kubernetes.io~projected/kube-api-access-2wdjl:{mountpoint:/var/lib/kubelet/pods/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba/volumes/kubernetes.io~projected/kube-api-access-2wdjl major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6662ac83-7fc2-451d-b26b-68d934badad1/volumes/kubernetes.io~projected/kube-api-access-nnb2q:{mountpoint:/var/lib/kubelet/pods/6662ac83-7fc2-451d-b26b-68d934badad1/volumes/kubernetes.io~projected/kube-api-access-nnb2q major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~projected/kube-api-access-fn87g:{mountpoint:/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~projected/kube-api-access-fn87g major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/693d053a-5b5c-4be2-9778-a4255c6f3417/volumes/kubernetes.io~projected/kube-api-access-hgh7v:{mountpoint:/var/lib/kubelet/pods/693d053a-5b5c-4be2-9778-a4255c6f3417/volumes/kubernetes.io~projected/kube-api-access-hgh7v major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~projected/kube-api-access-8s9mw:{mountpoint:/var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~projected/kube-api-access-8s9mw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85738300-00ba-4ba6-be20-2b259922fd7d/volumes/kubernetes.io~projected/kube-api-access-tqlhs:{mountpoint:/var/lib/kubelet/pods/85738300-00ba-4ba6-be20-2b259922fd7d/volumes/kubernetes.io~projected/kube-api-access-tqlhs major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c2ec818-97ad-4634-9c26-d1f6803d439d/volumes/kubernetes.io~projected/kube-api-access-xjrfb:{mountpoint:/var/lib/kubelet/pods/8c2ec818-97ad-4634-9c26-d1f6803d439d/volumes/kubernetes.io~projected/kube-api-access-xjrfb major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/929ccbc1-e13f-4dba-b00f-11a9c96d62f8/volumes/kubernetes.io~projected/kube-api-access-hlgxt:{mountpoint:/var/lib/kubelet/pods/929ccbc1-e13f-4dba-b00f-11a9c96d62f8/volumes/kubernetes.io~projected/kube-api-access-hlgxt major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/929ccbc1-e13f-4dba-b00f-11a9c96d62f8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/929ccbc1-e13f-4dba-b00f-11a9c96d62f8/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a04059dd-fc63-4523-829f-c0ce3c4b8129/volumes/kubernetes.io~projected/kube-api-access-xsspd:{mountpoint:/var/lib/kubelet/pods/a04059dd-fc63-4523-829f-c0ce3c4b8129/volumes/kubernetes.io~projected/kube-api-access-xsspd major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a04059dd-fc63-4523-829f-c0ce3c4b8129/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a04059dd-fc63-4523-829f-c0ce3c4b8129/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b02a9da2-ad91-4bac-8177-c851dc4a37d3/volumes/kubernetes.io~projected/kube-api-access-rk7sd:{mountpoint:/var/lib/kubelet/pods/b02a9da2-ad91-4bac-8177-c851dc4a37d3/volumes/kubernetes.io~projected/kube-api-access-rk7sd major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b60e34dd-a30c-4f35-9593-1935139f9a47/volumes/kubernetes.io~projected/kube-api-access-fdlb7:{mountpoint:/var/lib/kubelet/pods/b60e34dd-a30c-4f35-9593-1935139f9a47/volumes/kubernetes.io~projected/kube-api-access-fdlb7 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b60e34dd-a30c-4f35-9593-1935139f9a47/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b60e34dd-a30c-4f35-9593-1935139f9a47/volumes/kubernetes.io~secret/cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b97ac8d4-f132-4fa6-9f7f-d6d050098612/volumes/kubernetes.io~projected/kube-api-access-qqhms:{mountpoint:/var/lib/kubelet/pods/b97ac8d4-f132-4fa6-9f7f-d6d050098612/volumes/kubernetes.io~projected/kube-api-access-qqhms major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bddd1e71-4116-437a-b7ee-873bc218fa0b/volumes/kubernetes.io~projected/kube-api-access-cpkwr:{mountpoint:/var/lib/kubelet/pods/bddd1e71-4116-437a-b7ee-873bc218fa0b/volumes/kubernetes.io~projected/kube-api-access-cpkwr major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bddd1e71-4116-437a-b7ee-873bc218fa0b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bddd1e71-4116-437a-b7ee-873bc218fa0b/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0e41a51-8711-47dd-a04d-af098a4d7fcb/volumes/kubernetes.io~projected/kube-api-access-vv9vz:{mountpoint:/var/lib/kubelet/pods/c0e41a51-8711-47dd-a04d-af098a4d7fcb/volumes/kubernetes.io~projected/kube-api-access-vv9vz major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0e41a51-8711-47dd-a04d-af098a4d7fcb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c0e41a51-8711-47dd-a04d-af098a4d7fcb/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5bb30ba-a903-4c97-80a8-6a1f398b6664/volumes/kubernetes.io~projected/kube-api-access-pdsj7:{mountpoint:/var/lib/kubelet/pods/c5bb30ba-a903-4c97-80a8-6a1f398b6664/volumes/kubernetes.io~projected/kube-api-access-pdsj7 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4/volumes/kubernetes.io~projected/kube-api-access-vnvpv:{mountpoint:/var/lib/kubelet/pods/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4/volumes/kubernetes.io~projected/kube-api-access-vnvpv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f25010f1-45b5-4994-91fa-7de15f04fb08/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f25010f1-45b5-4994-91fa-7de15f04fb08/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volumes/kubernetes.io~projected/kube-api-access-sdptk:{mountpoint:/var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volumes/kubernetes.io~projected/kube-api-access-sdptk major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba6806a-680a-4fd0-8d5e-2b02db024040/volumes/kubernetes.io~projected/kube-api-access-kk4n2:{mountpoint:/var/lib/kubelet/pods/fba6806a-680a-4fd0-8d5e-2b02db024040/volumes/kubernetes.io~projected/kube-api-access-kk4n2 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~projected/kube-api-access-kr685:{mountpoint:/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~projected/kube-api-access-kr685 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20/volumes/kubernetes.io~projected/kube-api-access-zztc6:{mountpoint:/var/lib/kubelet/pods/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20/volumes/kubernetes.io~projected/kube-api-access-zztc6 major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/740ea2fc8780e067f2b1da9ab69e12792b7c6095843d143ae7b00857eed8c1e6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c9d45fa5ea9d850072189564884e5a12cc2baf047ce38e3c42a6b729a79a7e23/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/bf54bbeeb0880227c8915e8d12e3daa0ddb70508562208b7309cd6e30aaabd9d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9c45a8ae9d0e556ee17393a21b0cc098e16890b5358662f7c99d65e6ff398714/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2217d124e37265a47e4a3864c0306ddb179c89ebfacc0bb00034f2bbe1f24af5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bd478be4c27f6e9e40393aab6e32d23cbd16522489ef5374d2f48f6f08000760/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/61b79682f8ba17fc53786270dd4524878b0848ff7338cc6c9081d2827f299ed9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2a48ba0e60c8e674317ee1e757363c83a1a06b0d3b7b251b074630e332c4350e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/36bc6d9e5696a6cd8cbce6282f757de31f4f2d461fb549abc1c655071f15f024/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5601aababa09af8bb1bd5758b77cd54ecf7ca88dd3514f7acc8a7429413c8e55/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/fd936490202bfa035d7ab7cd526e50e8d8f54f801290197119bd80751539c580/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/93524932e1089e6e7324a769d22f07659cf15917d2ae791b9a94a67946128eda/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ce53102adbd209653807c3bc8eea93fdf0294bda98ce6dd2a3100ebeca6f56b2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5a7c9379640f5073017e5f3f67d8cc3ab5d5b74654fd85658e2d2c38e31cb0c5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d894b2a9011740ae39bca62dcdd672860f339f0965ced651c38189ede1fcadbf/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1d2d48ca338a88a8b8a7111338fa11e9c3153325a1b91b04739fd086f14caa7b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5f3c4e9f970f7ce27afa48fbd18e0a5803900a78463a3a90f7cf37c55e09ce8d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/c1a5dd798099eec3a6599bdfca284b83c67c8acab6a206c27e252fd7033b53c3/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b357484fce0692c2715050411777b35a5398f19bda6edb10e4a035ad0510a25a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/dda3464ca298baed56498b3202b976cad78c9fbe67d190532a8d9fb092215a78/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ad79936302084c4c548ed573be0ab16fcbbb6fd68f868bb837ddb1acf94725ff/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/95c9f3ef2d2fca0fe30141665a29a1291ed1c2ca3099476a402f09d0d6a23509/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b5693356b8c59136374af6b4ef31d03e3afe4a7e8b7be7c6e213afdcece3b480/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/0f9ec513c5c59874fb38801dc8ee32c03074d804cbe62729dce17f3b966f7ba8/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/63daf8f9b3438b5d269fb3c7937f5851794e45378be161080737ab97bd1205e3/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/6c607d8b4396b28a398d0e9cdd1a109bf2e962e96f3a3fdddbad540eea4ed966/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/5fbfe5905cc8729049b39b475206af68b6fa4d1aca0a6e380148b878487d1555/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/4fd6e1076cc24534753575398f9ff01e60a0c512e8a3db347e7d49ce70f5420c/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/d49a4132d2a8a59ead41e54d3d41b2c78ea55e61a4e6b2b8c9db170933c31cf4/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/6d2a7985d44b97c136a170495ff5ccc2640bee098b36294a5422dfe0def9f60d/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/3ac32a9a12c8f9d75df8b04eb110e99d999738e5a6463f0a8ede8e7d26a20b31/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/fee766d0caa227c7f304f61647490cf02ecc01c51ba336c4ce0f77c81e37e63a/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/bb59eedfa04e85e825bf1a99608e0901f6275986bda75ab55d052198d6fa840f/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/74f2cdca46adae1611a0060c38528127e315f884308660ce49d1ec40c6d6957a/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/f6fc65099cacaba2ed42fcd8a6971ea778a3ae1dd12d25de4bd7d99b85c799ec/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/aaa739424e3bae0a7e5595ab08a72870f694e233de138b1748636756e410091b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/29230d6ef5c503e4626de1199a50aebf7c5216955d7ad88a02339d73440c0205/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/106ed868cbb9fe2bc3882fe154f331a892806705310b4092c61f007f31d6d6d5/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/187e1d6deae9552b927830c99d741646f5be95e7bc009b8e1d7391f03fb89b5a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/913c8dbbee57fee074659fc47d77d57459d5004225a87af268be6a3eae1a1624/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/27d894438d42f95660605d2d20436ff4331f249c2561140b33ba5d570d68628e/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/04ad1c4dbdb781cbdae2f18fd9eea5bb12dae8c6f81f0acee0e73ecb10d32ea9/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/8ac37ca08fe7840ed07270cfd31c777ece638fec2aac2b52b2df9a972263af27/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/8526b39507135a1880324e18606ac654edb272980b74a1b856da02b2425020a7/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/70f08f7b6d8a7aa88d5e6b4bb05a71ed62830f2f3fe68151cb576faef6213ef2/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/e2a636bd541656fb55a311b3ca8daf03b649d53786355b56e987f80c27a42bf6/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/0de6de110109a918c4241aaa82d9de50abfe857d26b609f5ef360da875aa299b/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/edfa8984176a755dde5baad625398220e51d9759d3cfb9baa0f67e60a110d696/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/81b680bc18422c98507658c5a08b71af93499a2f490bcaf1daee1e9f22886680/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/d194cd8875355c8529ba2ca3b1388dc3402e4672c504b8834013a633697ce3fc/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/f2a2dd7d36f4cf0dc4019e793515731487249c3aa4bd2205096a8ac7afc5ddd8/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8be0ef9b313023f0809bea8e8b1b3c55269c0445d83b495526bd3cf6f692341f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/88c9faf150347cf7cd2060bf51e21dd072d406ebdcc9098f6ac76d84ba627fd8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/2c2b8d5a0cd3a40ea17d5c3b677fdf8f7830b46b7b4634ca0e014353bcfaf415/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f3f3030bd6fc9f138f07fa6f593ff84752d6ab955042d94cc1abaf6110fc2c90/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ad52010567b725405b591e6fcd1f477b34ea01b5120a72a79216f6152cdc111c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ed371368038d64a43fb92984b67157b97466922c04b7e2e53865644451263c49/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0f504981ecfc870ba2c3ef513b24e9514637382fe7e96a5c41b19485c9c2d678/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3d8f7cb915039b173a8458d992deafb6ab1beaeac770323f1d906b1f730704c6/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/faa79a02597225e2203ef0bd2a55b6bfe257a3f0b386b84aecdb3be80f8f3111/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.630807    4993 manager.go:217] Machine: {Timestamp:2026-10-30 12:32:15.630260751 +0000 UTC m=+0.105980355 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116276 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f6c7ad217886911c8954043c4229e SystemUUID:ec2f6c7a-d217-8869-11c8-954043c4229e BootID:d128c281-341a-4954-86ba-ddbc02a7acfd Filesystems:[{Device:/run/containers/storage/overlay-containers/e6783470b824e5ada08f8978159f8fa3ab14c870729c3a8ec8b483d07080ac13/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46fbee35-1719-4228-a543-874c0d3f4b4b/volumes/kubernetes.io~projected/kube-api-access-k5ng7 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539/volumes/kubernetes.io~projected/kube-api-access-4xknz DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73/volumes/kubernetes.io~projected/kube-api-access-65894 DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/720d4f499c17366e77d1fd53d62ad623a985e0656585df587066320bf011490a/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a04059dd-fc63-4523-829f-c0ce3c4b8129/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f06ee81df94126b4b32530df390aabe651063871419e45210003125f00ea1a7/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d03b27fdc9d8aad639bff0147a35d56b0f48865ed0901c0be9800cd94693aac7/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0463909a99b82c4211681b939877ecba262f152758f903a703ecff08af62b7a7/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bddd1e71-4116-437a-b7ee-873bc218fa0b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a04059dd-fc63-4523-829f-c0ce3c4b8129/volumes/kubernetes.io~projected/kube-api-access-xsspd DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c20bcc2117a1374ba4c1076841bfd38b29546d7898f5c7dfeb0401c850c8536/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad9cccb312fd7a735d377a7a0e2da0bc8bf993c3e5478c8e552116d748824203/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0e41a51-8711-47dd-a04d-af098a4d7fcb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c0c8c44da0c55454e6d8216104ddf1cf3a38822d1093e1449c0b94de4ab6b9c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/929ccbc1-e13f-4dba-b00f-11a9c96d62f8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fc05ca77bc319ed63fba43bf518d231b2de14d638da0b0c3424b238ea98c143/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9baf3572559e2aaf610fe99fac3cd83f6924b50d067dcacdbb27c86c5cc0925/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/b60e34dd-a30c-4f35-9593-1935139f9a47/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f25010f1-45b5-4994-91fa-7de15f04fb08/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fba6806a-680a-4fd0-8d5e-2b02db024040/volumes/kubernetes.io~projected/kube-api-access-kk4n2 DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17765d6d6bea4059edecf8e734b4903cc96402c65c441b1cf06c75418f0c78e5/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba/volumes/kubernetes.io~projected/kube-api-access-2wdjl DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdefe5619dd545d6d025a8459ccb04075fa6cee1ec873c1af5154d6a937f6c61/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~projected/kube-api-access-kr685 DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1a92d9fa3b304edb379f0e7e53cd25a8ef9e83dbaefff4c3bcf90e34404bc26/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/693d053a-5b5c-4be2-9778-a4255c6f3417/volumes/kubernetes.io~projected/kube-api-access-hgh7v DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2e18d6eb3287c9c812387f838104a490c223a3d411b4783dc04eddc90461540/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/179c5618-551b-4e6e-b0d3-5fa27708fb1f/volumes/kubernetes.io~projected/kube-api-access-5tmxt DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/090c5691-fa68-4427-84ed-e7b4797f77c7/volumes/kubernetes.io~projected/kube-api-access-gpjzd DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~projected/kube-api-access-fn87g DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0e41a51-8711-47dd-a04d-af098a4d7fcb/volumes/kubernetes.io~projected/kube-api-access-vv9vz DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/37d4d8e7-29c1-425a-87e9-d7fd23dce294/volumes/kubernetes.io~projected/kube-api-access-vtkrm DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc535eb9a5a55a3eba7ccdb414a6138ba0042246405d8cc38b264f3151e0ff33/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20/volumes/kubernetes.io~projected/kube-api-access-zztc6 DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d38ba665120444ee5af3c4a824bf416fab0acaa8aea94b94f66ddbcca61cad40/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd10a3ee2948075fd79c2f1fe32b86ca3c07f4dcb7c1f1434932193d68a91416/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b97ac8d4-f132-4fa6-9f7f-d6d050098612/volumes/kubernetes.io~projected/kube-api-access-qqhms DeviceMajor:0 DeviceMinor:323 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8670e2183233dad6028f9f37b60801aefef0cf2a99a0f93f76cab709b9537bdf/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6662ac83-7fc2-451d-b26b-68d934badad1/volumes/kubernetes.io~projected/kube-api-access-nnb2q DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b02a9da2-ad91-4bac-8177-c851dc4a37d3/volumes/kubernetes.io~projected/kube-api-access-rk7sd DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4/volumes/kubernetes.io~projected/kube-api-access-vnvpv DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c5bb30ba-a903-4c97-80a8-6a1f398b6664/volumes/kubernetes.io~projected/kube-api-access-pdsj7 DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b27de364aa5303a86c83db913102d4c4d79f2b4c36a4f69cf6c8d9e3e22ef31/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85738300-00ba-4ba6-be20-2b259922fd7d/volumes/kubernetes.io~projected/kube-api-access-tqlhs DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b60e34dd-a30c-4f35-9593-1935139f9a47/volumes/kubernetes.io~projected/kube-api-access-fdlb7 DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f81a461d86764aafb4034d3143211aa49bcbdbc2c8a146cc58b4e8cca73658a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66a6f187390b48b6d8e8f75b1155f773a9ed7a5f6666ffe3fa0cf9ca887f083f/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bddd1e71-4116-437a-b7ee-873bc218fa0b/volumes/kubernetes.io~projected/kube-api-access-cpkwr DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a226ce5143521ddba580e35aba3b0426a15d17a882263f5fb200c43d18d9e5b1/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/02e9f8c2-3da0-474f-b7db-40bee0e8d743/volumes/kubernetes.io~projected/kube-api-access-d8j7n DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1d64156d587787ba72c50535bb598061cdeacbf7c85d39df09c2995abacda57/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/45f0654b-10d6-4a98-b6f9-7b679ac028a2/volumes/kubernetes.io~projected/kube-api-access-j7rjf DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8c2ec818-97ad-4634-9c26-d1f6803d439d/volumes/kubernetes.io~projected/kube-api-access-xjrfb DeviceMajor:0 DeviceMinor:321 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f8924cae-fc43-49bd-9bab-d4bc56220d6a/volumes/kubernetes.io~projected/kube-api-access-sdptk DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3dd41889-9433-4fdb-8e05-52ea68e32536/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/51f4f667-c671-472c-ae68-beb10d17b645/volumes/kubernetes.io~projected/kube-api-access-kzq8v DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d15bf254ba722f58ad3aa21403defba71178ef5ceae362ece101745bfb77ce5/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/929ccbc1-e13f-4dba-b00f-11a9c96d62f8/volumes/kubernetes.io~projected/kube-api-access-hlgxt DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6af58857-77e1-4f65-89e0-3f7fa2ea3a52/volumes/kubernetes.io~projected/kube-api-access-8s9mw DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/090d2edb-0fb8-4fa0-8d91-d866c06226b3/volumes/kubernetes.io~projected/kube-api-access-dlt79 DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a633ffb31b7155f26758ce92d4c60b8e4ab7819b976eabdabbb98bf2e4c44458/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:17765d6d6bea405 MacAddress:ba:00:7d:fd:72:fe Speed:10000 Mtu:8901} {Name:1f06ee81df94126 MacAddress:7a:d7:2b:51:45:13 Speed:10000 Mtu:8901} {Name:1fc05ca77bc319e MacAddress:aa:9e:db:13:cf:35 Speed:10000 Mtu:8901} {Name:3d15bf254ba722f MacAddress:66:73:57:cb:ed:e3 Speed:10000 Mtu:8901} {Name:4f81a461d86764a MacAddress:66:d9:d1:9a:19:45 Speed:10000 Mtu:8901} {Name:66a6f187390b48b MacAddress:42:64:a9:73:2b:4a Speed:10000 Mtu:8901} {Name:720d4f499c17366 MacAddress:a2:17:f7:fc:e6:e4 Speed:10000 Mtu:8901} {Name:a1d64156d587787 MacAddress:8a:41:c9:b8:71:60 Speed:10000 Mtu:8901} {Name:a226ce5143521dd MacAddress:7e:31:9a:b8:92:e0 Speed:10000 Mtu:8901} {Name:a633ffb31b7155f MacAddress:02:11:df:c0:31:0a Speed:10000 Mtu:8901} {Name:a9baf3572559e2a MacAddress:d6:cf:b5:c0:70:ec Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d3:7a:d3:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:4a:1b:c5:26:1c:2a Speed:0 Mtu:8901} {Name:c2e18d6eb3287c9 MacAddress:ba:52:bc:3c:b0:74 Speed:10000 Mtu:8901} {Name:d03b27fdc9d8aad MacAddress:6e:67:ed:8a:5a:7a Speed:10000 Mtu:8901} {Name:dd10a3ee2948075 MacAddress:12:72:5f:d4:db:48 Speed:10000 Mtu:8901} {Name:e6783470b824e5a MacAddress:b6:2c:78:e8:55:c4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d3:7a:d3:55 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:a2:76:bc:a9:29:b7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:02:27:99:5b:2f:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631199    4993 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631266    4993 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631625    4993 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631648    4993 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-24.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631795    4993 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631803    4993 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631819    4993 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631832    4993 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.631853    4993 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.632009    4993 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.632123    4993 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.632160    4993 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.632170    4993 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.632182    4993 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.632193    4993 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.632214    4993 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.633102    4993 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.633114    4993 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:32:15.634614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.633780    4993 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:32:15.636618 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.636600    4993 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:32:15.636618 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.636617    4993 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:32:15.636746 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.636736    4993 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 12:32:15.636963 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.636950    4993 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:32:15.636993 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.636981    4993 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:32:15.637117 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637105    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:32:15.637151 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637128    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:32:15.637151 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637139    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:32:15.637151 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637148    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:32:15.637315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637157    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:32:15.637315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637167    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:32:15.637315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637173    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:32:15.637315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637180    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:32:15.637315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637187    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:32:15.637315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637193    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:32:15.637315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637302    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:32:15.637315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637313    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:32:15.637527 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637375    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:32:15.637527 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.637382    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:32:15.641336 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.641313    4993 server.go:1263] "Started kubelet"
Oct 30 12:32:15.641521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.641429    4993 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:32:15.641596 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.641551    4993 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:32:15.641934 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.641677    4993 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:32:15.642000 ip-10-0-135-24 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:32:15.642405 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.642121    4993 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:32:15.644306 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.644282    4993 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:32:15.647524 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.647500    4993 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:32:15.647911 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.647885    4993 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:32:15.648894 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.648267    4993 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:36 +0000 UTC" deadline="2028-05-16 12:34:11.924663241 +0000 UTC"
Oct 30 12:32:15.648894 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.648310    4993 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13536h1m56.276356956s"
Oct 30 12:32:15.648894 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.648609    4993 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:32:15.648894 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:15.648745    4993 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-24.ec2.internal\" not found"
Oct 30 12:32:15.649160 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.649060    4993 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:32:15.649160 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.649140    4993 factory.go:55] Registering systemd factory
Oct 30 12:32:15.649160 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.649158    4993 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:32:15.650717 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.650683    4993 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.651192    4993 factory.go:153] Registering CRI-O factory
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.651255    4993 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.651315    4993 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.651342    4993 factory.go:103] Registering Raw factory
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.651359    4993 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.651796    4993 manager.go:319] Starting recovery of all containers
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.653986    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a04059dd-fc63-4523-829f-c0ce3c4b8129" volumeName="kubernetes.io/configmap/a04059dd-fc63-4523-829f-c0ce3c4b8129-config" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654027    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bddd1e71-4116-437a-b7ee-873bc218fa0b" volumeName="kubernetes.io/secret/bddd1e71-4116-437a-b7ee-873bc218fa0b-serving-cert" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654041    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0e41a51-8711-47dd-a04d-af098a4d7fcb" volumeName="kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-config" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654054    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f4f667-c671-472c-ae68-beb10d17b645" volumeName="kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-stats-auth" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654067    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dd41889-9433-4fdb-8e05-52ea68e32536" volumeName="kubernetes.io/configmap/3dd41889-9433-4fdb-8e05-52ea68e32536-konnectivity-ca" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654079    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46fbee35-1719-4228-a543-874c0d3f4b4b" volumeName="kubernetes.io/configmap/46fbee35-1719-4228-a543-874c0d3f4b4b-serviceca" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654092    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af58857-77e1-4f65-89e0-3f7fa2ea3a52" volumeName="kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-tuned" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654104    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af58857-77e1-4f65-89e0-3f7fa2ea3a52" volumeName="kubernetes.io/projected/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-kube-api-access-8s9mw" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654118    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8924cae-fc43-49bd-9bab-d4bc56220d6a" volumeName="kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-config" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654130    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcbd57e7-45d9-472f-9afe-ded1285276b9" volumeName="kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654143    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b3f7eca-7b44-46c3-b1fd-d9bb4f848539" volumeName="kubernetes.io/projected/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539-kube-api-access-4xknz" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654157    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="179c5618-551b-4e6e-b0d3-5fa27708fb1f" volumeName="kubernetes.io/projected/179c5618-551b-4e6e-b0d3-5fa27708fb1f-kube-api-access-5tmxt" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654169    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67405c0f-c406-4e55-ade6-ba7a26a99a38" volumeName="kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654188    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67405c0f-c406-4e55-ade6-ba7a26a99a38" volumeName="kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654222    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67405c0f-c406-4e55-ade6-ba7a26a99a38" volumeName="kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654236    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4" volumeName="kubernetes.io/projected/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-kube-api-access-vnvpv" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654247    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8924cae-fc43-49bd-9bab-d4bc56220d6a" volumeName="kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654259    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8924cae-fc43-49bd-9bab-d4bc56220d6a" volumeName="kubernetes.io/secret/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654271    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b97ac8d4-f132-4fa6-9f7f-d6d050098612" volumeName="kubernetes.io/projected/b97ac8d4-f132-4fa6-9f7f-d6d050098612-kube-api-access-qqhms" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654300    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0e41a51-8711-47dd-a04d-af098a4d7fcb" volumeName="kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-trusted-ca" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654312    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcbd57e7-45d9-472f-9afe-ded1285276b9" volumeName="kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654323    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcbd57e7-45d9-472f-9afe-ded1285276b9" volumeName="kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654336    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="061ed4c7-604d-4cea-aedf-f7bbbb7a1c73" volumeName="kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654348    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6662ac83-7fc2-451d-b26b-68d934badad1" volumeName="kubernetes.io/projected/6662ac83-7fc2-451d-b26b-68d934badad1-kube-api-access-nnb2q" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654362    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="693d053a-5b5c-4be2-9778-a4255c6f3417" volumeName="kubernetes.io/configmap/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-config" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654376    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0e41a51-8711-47dd-a04d-af098a4d7fcb" volumeName="kubernetes.io/projected/c0e41a51-8711-47dd-a04d-af098a4d7fcb-kube-api-access-vv9vz" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654389    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0e41a51-8711-47dd-a04d-af098a4d7fcb" volumeName="kubernetes.io/secret/c0e41a51-8711-47dd-a04d-af098a4d7fcb-serving-cert" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654402    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8924cae-fc43-49bd-9bab-d4bc56220d6a" volumeName="kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-env-overrides" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654442    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" volumeName="kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654476    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d6d13bc-3169-4d2c-830c-66837db3faf4" volumeName="kubernetes.io/configmap/5d6d13bc-3169-4d2c-830c-66837db3faf4-nginx-conf" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654494    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02e9f8c2-3da0-474f-b7db-40bee0e8d743" volumeName="kubernetes.io/projected/02e9f8c2-3da0-474f-b7db-40bee0e8d743-kube-api-access-d8j7n" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654511    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c2ec818-97ad-4634-9c26-d1f6803d439d" volumeName="kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654542    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c2ec818-97ad-4634-9c26-d1f6803d439d" volumeName="kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654555    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcbd57e7-45d9-472f-9afe-ded1285276b9" volumeName="kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654568    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02e9f8c2-3da0-474f-b7db-40bee0e8d743" volumeName="kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654580    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="929ccbc1-e13f-4dba-b00f-11a9c96d62f8" volumeName="kubernetes.io/projected/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-kube-api-access-hlgxt" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654592    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b60e34dd-a30c-4f35-9593-1935139f9a47" volumeName="kubernetes.io/secret/b60e34dd-a30c-4f35-9593-1935139f9a47-cert" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654610    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25010f1-45b5-4994-91fa-7de15f04fb08" volumeName="kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654624    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8924cae-fc43-49bd-9bab-d4bc56220d6a" volumeName="kubernetes.io/projected/f8924cae-fc43-49bd-9bab-d4bc56220d6a-kube-api-access-sdptk" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654637    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45f0654b-10d6-4a98-b6f9-7b679ac028a2" volumeName="kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654650    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" volumeName="kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654663    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af58857-77e1-4f65-89e0-3f7fa2ea3a52" volumeName="kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-tmp" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654676    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f4f667-c671-472c-ae68-beb10d17b645" volumeName="kubernetes.io/projected/51f4f667-c671-472c-ae68-beb10d17b645-kube-api-access-kzq8v" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654692    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c2ec818-97ad-4634-9c26-d1f6803d439d" volumeName="kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654705    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="929ccbc1-e13f-4dba-b00f-11a9c96d62f8" volumeName="kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-tmp" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654720    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b02a9da2-ad91-4bac-8177-c851dc4a37d3" volumeName="kubernetes.io/projected/b02a9da2-ad91-4bac-8177-c851dc4a37d3-kube-api-access-rk7sd" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654734    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b60e34dd-a30c-4f35-9593-1935139f9a47" volumeName="kubernetes.io/projected/b60e34dd-a30c-4f35-9593-1935139f9a47-kube-api-access-fdlb7" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654755    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="179c5618-551b-4e6e-b0d3-5fa27708fb1f" volumeName="kubernetes.io/configmap/179c5618-551b-4e6e-b0d3-5fa27708fb1f-config-volume" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654770    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45f0654b-10d6-4a98-b6f9-7b679ac028a2" volumeName="kubernetes.io/empty-dir/45f0654b-10d6-4a98-b6f9-7b679ac028a2-data-volume" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654787    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="929ccbc1-e13f-4dba-b00f-11a9c96d62f8" volumeName="kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-service-ca-bundle" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654802    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5bb30ba-a903-4c97-80a8-6a1f398b6664" volumeName="kubernetes.io/projected/c5bb30ba-a903-4c97-80a8-6a1f398b6664-kube-api-access-pdsj7" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654815    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" volumeName="kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654829    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67405c0f-c406-4e55-ade6-ba7a26a99a38" volumeName="kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654844    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bddd1e71-4116-437a-b7ee-873bc218fa0b" volumeName="kubernetes.io/configmap/bddd1e71-4116-437a-b7ee-873bc218fa0b-config" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654857    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bddd1e71-4116-437a-b7ee-873bc218fa0b" volumeName="kubernetes.io/projected/bddd1e71-4116-437a-b7ee-873bc218fa0b-kube-api-access-cpkwr" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654873    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4" volumeName="kubernetes.io/empty-dir/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-tmp-dir" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654913    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcbd57e7-45d9-472f-9afe-ded1285276b9" volumeName="kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654926    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02e9f8c2-3da0-474f-b7db-40bee0e8d743" volumeName="kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-daemon-config" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654938    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37d4d8e7-29c1-425a-87e9-d7fd23dce294" volumeName="kubernetes.io/projected/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kube-api-access-vtkrm" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654949    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67405c0f-c406-4e55-ade6-ba7a26a99a38" volumeName="kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654960    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67405c0f-c406-4e55-ade6-ba7a26a99a38" volumeName="kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654973    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="929ccbc1-e13f-4dba-b00f-11a9c96d62f8" volumeName="kubernetes.io/secret/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-serving-cert" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654984    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcbd57e7-45d9-472f-9afe-ded1285276b9" volumeName="kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.654996    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20" volumeName="kubernetes.io/projected/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-kube-api-access-zztc6" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655008    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45f0654b-10d6-4a98-b6f9-7b679ac028a2" volumeName="kubernetes.io/projected/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-api-access-j7rjf" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655020    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46fbee35-1719-4228-a543-874c0d3f4b4b" volumeName="kubernetes.io/projected/46fbee35-1719-4228-a543-874c0d3f4b4b-kube-api-access-k5ng7" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655033    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67405c0f-c406-4e55-ade6-ba7a26a99a38" volumeName="kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655045    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="929ccbc1-e13f-4dba-b00f-11a9c96d62f8" volumeName="kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655057    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a04059dd-fc63-4523-829f-c0ce3c4b8129" volumeName="kubernetes.io/projected/a04059dd-fc63-4523-829f-c0ce3c4b8129-kube-api-access-xsspd" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655069    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="090c5691-fa68-4427-84ed-e7b4797f77c7" volumeName="kubernetes.io/projected/090c5691-fa68-4427-84ed-e7b4797f77c7-kube-api-access-gpjzd" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655080    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="179c5618-551b-4e6e-b0d3-5fa27708fb1f" volumeName="kubernetes.io/empty-dir/179c5618-551b-4e6e-b0d3-5fa27708fb1f-tmp-dir" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655092    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dd41889-9433-4fdb-8e05-52ea68e32536" volumeName="kubernetes.io/secret/3dd41889-9433-4fdb-8e05-52ea68e32536-agent-certs" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655106    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="929ccbc1-e13f-4dba-b00f-11a9c96d62f8" volumeName="kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-snapshots" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655119    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="090d2edb-0fb8-4fa0-8d91-d866c06226b3" volumeName="kubernetes.io/projected/090d2edb-0fb8-4fa0-8d91-d866c06226b3-kube-api-access-dlt79" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655132    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f4f667-c671-472c-ae68-beb10d17b645" volumeName="kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-default-certificate" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655144    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85738300-00ba-4ba6-be20-2b259922fd7d" volumeName="kubernetes.io/configmap/85738300-00ba-4ba6-be20-2b259922fd7d-iptables-alerter-script" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655155    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c2ec818-97ad-4634-9c26-d1f6803d439d" volumeName="kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655168    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a04059dd-fc63-4523-829f-c0ce3c4b8129" volumeName="kubernetes.io/secret/a04059dd-fc63-4523-829f-c0ce3c4b8129-serving-cert" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655181    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcbd57e7-45d9-472f-9afe-ded1285276b9" volumeName="kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655195    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20" volumeName="kubernetes.io/configmap/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-telemetry-config" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655222    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" volumeName="kubernetes.io/projected/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-kube-api-access-2wdjl" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655236    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba6806a-680a-4fd0-8d5e-2b02db024040" volumeName="kubernetes.io/projected/fba6806a-680a-4fd0-8d5e-2b02db024040-kube-api-access-kk4n2" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655249    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="693d053a-5b5c-4be2-9778-a4255c6f3417" volumeName="kubernetes.io/projected/693d053a-5b5c-4be2-9778-a4255c6f3417-kube-api-access-hgh7v" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655261    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85738300-00ba-4ba6-be20-2b259922fd7d" volumeName="kubernetes.io/projected/85738300-00ba-4ba6-be20-2b259922fd7d-kube-api-access-tqlhs" seLinuxMountContext=""
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655272    4993 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:32:15.657248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.655280    4993 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:32:15.673022 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.672995    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/1.log"
Oct 30 12:32:15.673022 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.673021    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/1.log"
Oct 30 12:32:15.681467 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.681425    4993 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:32:15.683165 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.683136    4993 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:32:15.683342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.683225    4993 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:32:15.683342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.683252    4993 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:32:15.683342 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:15.683297    4993 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 12:32:15.700262 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.700236    4993 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 12:32:15.702073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.702045    4993 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:32:15.712489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.712460    4993 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:32:15.714245 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.714200    4993 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:32:15.714711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.714687    4993 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:32:15.729440 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.729395    4993 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="6633d2d70e91016edd5f03b7eb154d83b0e079126763b2fc69a4a341e7a0c885" exitCode=0
Oct 30 12:32:15.729440 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.729436    4993 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="25f30b5b25b53396cb3bb0f352696f3cc50574abf7c0f8be786ae34bd40c15de" exitCode=0
Oct 30 12:32:15.729666 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.729449    4993 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="87304e5d2a4e705f5751d8d6d965a34f6fd35ee6ceff3e0552727ff6f46900c0" exitCode=0
Oct 30 12:32:15.729666 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.729460    4993 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="cf8d1ba34c7e61259abf65fa23ee245bf3e8113dbe3b9571c4d7e8c109fd5163" exitCode=0
Oct 30 12:32:15.729666 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.729473    4993 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="d721300d72a165702116a73239e41434d1600a72a962eed4e1b8721f21580cb6" exitCode=0
Oct 30 12:32:15.729666 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.729482    4993 generic.go:356] "Generic (PLEG): container finished" podID="53f78bb5-5e15-4574-93d6-9cdbdc2c95ba" containerID="f1508d34142b4a5603b32b472a98c59141845677b94bd28aec017c7149ac913f" exitCode=0
Oct 30 12:32:15.736703 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.736671    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/1.log"
Oct 30 12:32:15.736889 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.736712    4993 generic.go:356] "Generic (PLEG): container finished" podID="c0e41a51-8711-47dd-a04d-af098a4d7fcb" containerID="5456cdf1ef89b8c177fbf99cbd62f12e9886cfd5ae61fdd92c2309066c57513a" exitCode=255
Oct 30 12:32:15.743339 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.743284    4993 generic.go:356] "Generic (PLEG): container finished" podID="7813451d010c378fca54e5b82e788cc7" containerID="bb8fe266f5f25173304cfb5935a276af4203d6fcad9ffa6207ccdbd11d05c962" exitCode=0
Oct 30 12:32:15.755345 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.755310    4993 manager.go:324] Recovery completed
Oct 30 12:32:15.777631 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.777597    4993 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:32:15.777631 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.777625    4993 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:32:15.777886 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.777669    4993 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:32:15.777953 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.777931    4993 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 12:32:15.778004 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.777956    4993 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 12:32:15.778061 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.778006    4993 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 12:32:15.778061 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.778016    4993 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 12:32:15.778061 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.778025    4993 policy_none.go:50] "Start"
Oct 30 12:32:15.778200 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.778062    4993 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:32:15.778200 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.778078    4993 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:32:15.778422 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.778407    4993 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 12:32:15.778495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.778424    4993 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 12:32:15.778495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.778433    4993 policy_none.go:44] "Start"
Oct 30 12:32:15.783704 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:15.783672    4993 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 12:32:15.783843 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.783818    4993 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:32:15.786476 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.786456    4993 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:32:15.787016 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:15.786491    4993 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:32:15.787103 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.787019    4993 server.go:87] "Starting device plugin registration server"
Oct 30 12:32:15.787359 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.787344    4993 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:32:15.787442 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.787359    4993 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:32:15.787490 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.787464    4993 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:32:15.787637 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.787595    4993 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:32:15.787755 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.787642    4993 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:32:15.787755 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.787648    4993 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:32:15.787851 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.787707    4993 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:32:15.787631705Z","UUID":"6170396a-a3f1-43b1-976b-2a8e3065127e","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:32:15.789109 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.789084    4993 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:32:15.789224 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.789117    4993 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:32:15.789224 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:15.789127    4993 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:32:15.888374 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.888320    4993 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:32:15.889727 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.889707    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:32:15.889842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.889740    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:32:15.889842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.889755    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-24.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:32:15.889842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.889812    4993 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-24.ec2.internal"
Oct 30 12:32:15.925402 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.925364    4993 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-24.ec2.internal"
Oct 30 12:32:15.925591 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.925477    4993 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-24.ec2.internal"
Oct 30 12:32:15.984835 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.984756    4993 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"]
Oct 30 12:32:15.985470 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.985434    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal" event={"ID":"fdd80fd13fc4b06ccce2af5848f9d942","Type":"ContainerStarted","Data":"3661a566248c266c90bfbc677c65b754a7bfa085b8c06021d656d97528e6a781"}
Oct 30 12:32:15.985522 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.985473    4993 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:15.985566 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.985477    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal" event={"ID":"fdd80fd13fc4b06ccce2af5848f9d942","Type":"ContainerStarted","Data":"8670e2183233dad6028f9f37b60801aefef0cf2a99a0f93f76cab709b9537bdf"}
Oct 30 12:32:15.985655 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.985643    4993 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:15.985688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.985661    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal" event={"ID":"7813451d010c378fca54e5b82e788cc7","Type":"ContainerStarted","Data":"e829ade0b8d4f96be97b96d49c1d2646250846470b31f1cace6f59950288d1b4"}
Oct 30 12:32:15.985718 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.985697    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal" event={"ID":"7813451d010c378fca54e5b82e788cc7","Type":"ContainerDied","Data":"bb8fe266f5f25173304cfb5935a276af4203d6fcad9ffa6207ccdbd11d05c962"}
Oct 30 12:32:15.985768 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:15.985716    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal" event={"ID":"7813451d010c378fca54e5b82e788cc7","Type":"ContainerStarted","Data":"9b27de364aa5303a86c83db913102d4c4d79f2b4c36a4f69cf6c8d9e3e22ef31"}
Oct 30 12:32:16.009812 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.009779    4993 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:32:16.009986 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.009839    4993 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-24.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.026408 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.026372    4993 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:32:16.026561 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.026434    4993 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.101659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.101618    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdd80fd13fc4b06ccce2af5848f9d942-config\") pod \"kube-apiserver-proxy-ip-10-0-135-24.ec2.internal\" (UID: \"fdd80fd13fc4b06ccce2af5848f9d942\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.101659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.101657    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.101870 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.101676    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.202399 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.202294    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.202399 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.202348    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdd80fd13fc4b06ccce2af5848f9d942-config\") pod \"kube-apiserver-proxy-ip-10-0-135-24.ec2.internal\" (UID: \"fdd80fd13fc4b06ccce2af5848f9d942\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.202399 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.202370    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.202621 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.202410    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.202621 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.202414    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7813451d010c378fca54e5b82e788cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" (UID: \"7813451d010c378fca54e5b82e788cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.202621 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.202414    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdd80fd13fc4b06ccce2af5848f9d942-config\") pod \"kube-apiserver-proxy-ip-10-0-135-24.ec2.internal\" (UID: \"fdd80fd13fc4b06ccce2af5848f9d942\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.633074 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.633035    4993 apiserver.go:52] "Watching apiserver"
Oct 30 12:32:16.687317 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.687287    4993 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:32:16.688049 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.688017    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vd2n6","openshift-network-operator/iptables-alerter-4mvs6","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs","openshift-insights/periodic-gathering-zks2p-9688t","openshift-network-console/networking-console-plugin-86c478bb88-87dsg","openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6","openshift-insights/insights-runtime-extractor-gl9mb","openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q","kube-system/global-pull-secret-syncer-s4gm9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp","kube-system/konnectivity-agent-tnrf7","openshift-image-registry/node-ca-wjpmw","kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft","openshift-multus/multus-additional-cni-plugins-n6hfj","openshift-multus/network-metrics-daemon-7dxkx","openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k","kube-system/kas-connection-checker-685895c7c9-829vl","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb","openshift-ingress/router-default-77bb4759d7-ztvnj","openshift-insights/insights-operator-7c4986d449-7tvl5","openshift-cluster-node-tuning-operator/tuned-sq7sj","openshift-dns/dns-default-qngfl","openshift-dns/node-resolver-98w6d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal","openshift-network-diagnostics/network-check-target-fkhv4","openshift-ovn-kubernetes/ovnkube-node-svlgw","openshift-image-registry/image-registry-6676c6bb89-m66tl","openshift-multus/multus-89m26","openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl","kube-system/kas-connection-checker-685895c7c9-vwdb5","kube-system/kas-connection-checker-685895c7c9-pbb9n","openshift-image-registry/image-registry-6ccff9c748-92chp","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx","openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"]
Oct 30 12:32:16.689262 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.689240    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:16.690159 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.690138    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:16.690475 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.690453    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.691231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.691196    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.692836 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.692817    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:16.700920 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.700887    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:16.702592 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.702570    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.703592 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.703568    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.704083 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704056    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.704262 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704059    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.704262 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704066    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.704262 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704119    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 12:32:16.704262 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704157    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 12:32:16.704262 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704159    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 12:32:16.704499 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704261    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.704499 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704313    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.704499 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.704381    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pwb6k\""
Oct 30 12:32:16.719337 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.719295    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 12:32:16.719538 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.719368    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.719538 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.719438    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.719538 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.719449    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.719538 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.719460    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.720658 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.720630    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 12:32:16.724823 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.724763    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.725082 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725044    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.725082 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725048    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.725082 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725070    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.725336 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725118    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:32:16.725336 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725121    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 12:32:16.725336 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725153    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.725541 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725366    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.725541 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725380    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.725541 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725504    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:32:16.725779 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725765    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.725893 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.725878    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.726291 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.726277    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:32:16.726343 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.726284    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:32:16.726343 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.726301    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:32:16.726423 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.726408    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.732383 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732361    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 12:32:16.732383 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732361    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:32:16.732620 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732386    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 12:32:16.732620 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732425    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.732620 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732435    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.732620 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732496    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:32:16.732827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732662    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.732827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732700    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:32:16.732827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732704    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:32:16.732827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732751    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732872    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732914    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732950    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732969    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.732997    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.733005    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.733050    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.733082    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.733106    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.733147    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.733197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.733270    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.733961 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.733360    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.733961 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.733685    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.734257 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734236    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 12:32:16.734257 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734251    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 12:32:16.734387 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734281    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:32:16.734387 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734317    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:16.734387 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734342    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:32:16.734493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734389    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 12:32:16.734493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734465    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 12:32:16.734493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734465    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 12:32:16.734600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734513    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 12:32:16.734600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734531    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 12:32:16.734600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734543    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.734600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734546    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:32:16.734600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734565    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 12:32:16.734600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734585    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734608    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734633    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734643    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734637    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734673    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734702    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734703    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734743    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rclv\""
Oct 30 12:32:16.734888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.734750    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtx2d\""
Oct 30 12:32:16.735289 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.735269    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lcssw\""
Oct 30 12:32:16.735365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.735313    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6btbg\""
Oct 30 12:32:16.735365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.735353    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zxlk2\""
Oct 30 12:32:16.735589 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.735574    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.735646 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.735607    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rszcr\""
Oct 30 12:32:16.735829 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.735815    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.736071 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.736055    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:16.737404 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.737382    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:32:16.739342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739309    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mtq9t\""
Oct 30 12:32:16.739760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739737    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wqghl\""
Oct 30 12:32:16.739838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739763    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kq49t\""
Oct 30 12:32:16.739838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739782    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tpkdl\""
Oct 30 12:32:16.739838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739791    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:32:16.739838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739750    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxtxl\""
Oct 30 12:32:16.739838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739749    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgbt6\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739882    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5fvrw\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739897    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k87jk\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739902    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739917    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5lgc6\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739890    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xngpq\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739927    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pmx9z\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739946    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56d7b\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739970    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sr4bw\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.739978    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zprfc\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.740017    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fkzfg\""
Oct 30 12:32:16.740079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.740079    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-g89j4\""
Oct 30 12:32:16.740491 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.740088    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:32:16.740491 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.740093    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5xhj6\""
Oct 30 12:32:16.740491 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.740115    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7nlfh\""
Oct 30 12:32:16.740491 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.740153    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bklll\""
Oct 30 12:32:16.740650 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.740630    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v9g2h\""
Oct 30 12:32:16.745221 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.745174    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 12:32:16.745752 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.745732    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 12:32:16.749881 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.749854    4993 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.750037 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.750022    4993 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.753176 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.753151    4993 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:32:16.773408 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.773376    4993 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:32:16.773543 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.773436    4993 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-24.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.773895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.773881    4993 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:32:16.773931 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.773915    4993 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-24.ec2.internal"
Oct 30 12:32:16.798960 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.798930    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vd2n6_b60e34dd-a30c-4f35-9593-1935139f9a47/serve-healthcheck-canary/0.log"
Oct 30 12:32:16.804071 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804040    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-etc-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.804190 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804085    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ng7\" (UniqueName: \"kubernetes.io/projected/46fbee35-1719-4228-a543-874c0d3f4b4b-kube-api-access-k5ng7\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:32:16.804190 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804104    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5d6d13bc-3169-4d2c-830c-66837db3faf4-nginx-conf\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:16.804190 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804162    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:16.804351 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804202    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cni-binary-copy\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.804351 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804256    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-socket-dir-parent\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.804447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804371    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.804447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804405    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-config\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:16.804447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804405    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5d6d13bc-3169-4d2c-830c-66837db3faf4-nginx-conf\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:16.804570 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804465    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysconfig\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.804570 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804542    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:16.804643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804577    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46fbee35-1719-4228-a543-874c0d3f4b4b-serviceca\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:32:16.804643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804609    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-slash\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.804643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804617    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cni-binary-copy\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.804643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804636    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.804798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804674    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/179c5618-551b-4e6e-b0d3-5fa27708fb1f-config-volume\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.804798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804726    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a04059dd-fc63-4523-829f-c0ce3c4b8129-serving-cert\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:16.804798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804761    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-config\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:16.804798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804776    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804817    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46fbee35-1719-4228-a543-874c0d3f4b4b-serviceca\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804818    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179c5618-551b-4e6e-b0d3-5fa27708fb1f-tmp-dir\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804854    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804857    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/179c5618-551b-4e6e-b0d3-5fa27708fb1f-config-volume\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804881    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179c5618-551b-4e6e-b0d3-5fa27708fb1f-tmp-dir\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804881    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804910    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45f0654b-10d6-4a98-b6f9-7b679ac028a2-crio-socket\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804933    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804963    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-systemd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.804977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.804971    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a04059dd-fc63-4523-829f-c0ce3c4b8129-serving-cert\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805000    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovn-node-metrics-cert\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805060    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805100    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805146    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjzd\" (UniqueName: \"kubernetes.io/projected/090c5691-fa68-4427-84ed-e7b4797f77c7-kube-api-access-gpjzd\") pod \"network-check-source-6f78d59f99-fc5ft\" (UID: \"090c5691-fa68-4427-84ed-e7b4797f77c7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805187    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-kubelet-config\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805188    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805273    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdsj7\" (UniqueName: \"kubernetes.io/projected/c5bb30ba-a903-4c97-80a8-6a1f398b6664-kube-api-access-pdsj7\") pod \"kas-connection-checker-685895c7c9-pbb9n\" (UID: \"c5bb30ba-a903-4c97-80a8-6a1f398b6664\") " pod="kube-system/kas-connection-checker-685895c7c9-pbb9n"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805315    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805353    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt79\" (UniqueName: \"kubernetes.io/projected/090d2edb-0fb8-4fa0-8d91-d866c06226b3-kube-api-access-dlt79\") pod \"kas-connection-checker-685895c7c9-vwdb5\" (UID: \"090d2edb-0fb8-4fa0-8d91-d866c06226b3\") " pod="kube-system/kas-connection-checker-685895c7c9-vwdb5"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805386    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovn-node-metrics-cert\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805433    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7sd\" (UniqueName: \"kubernetes.io/projected/b02a9da2-ad91-4bac-8177-c851dc4a37d3-kube-api-access-rk7sd\") pod \"volume-data-source-validator-58c9f5f46d-4t2tb\" (UID: \"b02a9da2-ad91-4bac-8177-c851dc4a37d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805474    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.805508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805507    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805580    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805612    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-default-certificate\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805668    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805699    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805705    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85738300-00ba-4ba6-be20-2b259922fd7d-iptables-alerter-script\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805736    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-systemd\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805784    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-bin\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805836    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805911    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-kubernetes\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805957    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-tmp-dir\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805972    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-default-certificate\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.805991    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.806050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806022    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85738300-00ba-4ba6-be20-2b259922fd7d-iptables-alerter-script\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806065    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-tmp-dir\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806077    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-registration-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806115    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-lib-modules\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806148    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806153    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztc6\" (UniqueName: \"kubernetes.io/projected/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-kube-api-access-zztc6\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806192    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvpv\" (UniqueName: \"kubernetes.io/projected/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-kube-api-access-vnvpv\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806316    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-node-log\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806389    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-sys\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806427    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60e34dd-a30c-4f35-9593-1935139f9a47-cert\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806442    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806456    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4n2\" (UniqueName: \"kubernetes.io/projected/fba6806a-680a-4fd0-8d5e-2b02db024040-kube-api-access-kk4n2\") pod \"migrator-66df5d496b-s9s6q\" (UID: \"fba6806a-680a-4fd0-8d5e-2b02db024040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806484    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-serving-cert\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806559    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fbee35-1719-4228-a543-874c0d3f4b4b-host\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:32:16.806579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806580    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8j7n\" (UniqueName: \"kubernetes.io/projected/02e9f8c2-3da0-474f-b7db-40bee0e8d743-kube-api-access-d8j7n\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806601    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdjl\" (UniqueName: \"kubernetes.io/projected/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-kube-api-access-2wdjl\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806624    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806696    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-serving-cert\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806726    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b60e34dd-a30c-4f35-9593-1935139f9a47-cert\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806848    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xknz\" (UniqueName: \"kubernetes.io/projected/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539-kube-api-access-4xknz\") pod \"kas-connection-checker-685895c7c9-829vl\" (UID: \"0b3f7eca-7b44-46c3-b1fd-d9bb4f848539\") " pod="kube-system/kas-connection-checker-685895c7c9-829vl"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806885    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh7v\" (UniqueName: \"kubernetes.io/projected/693d053a-5b5c-4be2-9778-a4255c6f3417-kube-api-access-hgh7v\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806906    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cnibin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806911    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806925    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.806946    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-ovn\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.807113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807096    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzq8v\" (UniqueName: \"kubernetes.io/projected/51f4f667-c671-472c-ae68-beb10d17b645-kube-api-access-kzq8v\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807126    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807156    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-system-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807175    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-netns\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807193    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-etc-kubernetes\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807224    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0654b-10d6-4a98-b6f9-7b679ac028a2-data-volume\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807255    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bddd1e71-4116-437a-b7ee-873bc218fa0b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807424    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-env-overrides\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807452    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45f0654b-10d6-4a98-b6f9-7b679ac028a2-data-volume\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807472    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-sys-fs\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807495    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807526    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.807577 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807571    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807629    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807647    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807663    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-env-overrides\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807681    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-hosts-file\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807728    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkrm\" (UniqueName: \"kubernetes.io/projected/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kube-api-access-vtkrm\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807737    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807763    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-conf\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807792    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bddd1e71-4116-437a-b7ee-873bc218fa0b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807794    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807833    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807889    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807909    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd41889-9433-4fdb-8e05-52ea68e32536-konnectivity-ca\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807928    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkwr\" (UniqueName: \"kubernetes.io/projected/bddd1e71-4116-437a-b7ee-873bc218fa0b-kube-api-access-cpkwr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807965    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807966    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807981    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807971    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.807999    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.808129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808124    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808173    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808217    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-script-lib\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808246    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlhs\" (UniqueName: \"kubernetes.io/projected/85738300-00ba-4ba6-be20-2b259922fd7d-kube-api-access-tqlhs\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808264    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e41a51-8711-47dd-a04d-af098a4d7fcb-serving-cert\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808280    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-host\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808329    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlgxt\" (UniqueName: \"kubernetes.io/projected/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-kube-api-access-hlgxt\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808354    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808393    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dd41889-9433-4fdb-8e05-52ea68e32536-konnectivity-ca\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808403    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-script-lib\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808438    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-k8s-cni-cncf-io\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808520    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-netns\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808533    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e41a51-8711-47dd-a04d-af098a4d7fcb-serving-cert\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808542    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnb2q\" (UniqueName: \"kubernetes.io/projected/6662ac83-7fc2-451d-b26b-68d934badad1-kube-api-access-nnb2q\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808579    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-stats-auth\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808660    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-tmp\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808758    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-run\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808775    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-tmp\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808794    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn87g\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808816    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-stats-auth\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808822    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhms\" (UniqueName: \"kubernetes.io/projected/b97ac8d4-f132-4fa6-9f7f-d6d050098612-kube-api-access-qqhms\") pod \"migrator-66df5d496b-zl9k6\" (UID: \"b97ac8d4-f132-4fa6-9f7f-d6d050098612\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808867    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-snapshots\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.808908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808896    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808939    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-snapshots\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808939    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808974    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-modprobe-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.808993    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809012    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-hostroot\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809054    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsspd\" (UniqueName: \"kubernetes.io/projected/a04059dd-fc63-4523-829f-c0ce3c4b8129-kube-api-access-xsspd\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809082    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9mw\" (UniqueName: \"kubernetes.io/projected/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-kube-api-access-8s9mw\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809098    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-dbus\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809124    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809126    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809182    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809231    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809259    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rjf\" (UniqueName: \"kubernetes.io/projected/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-api-access-j7rjf\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809284    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-kubelet\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809279    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809314    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmxt\" (UniqueName: \"kubernetes.io/projected/179c5618-551b-4e6e-b0d3-5fa27708fb1f-kube-api-access-5tmxt\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809401    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809432    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809459    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-socket-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809494    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809519    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd41889-9433-4fdb-8e05-52ea68e32536-agent-certs\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809542    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809558    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-var-lib-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809596    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-config\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809621    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrfb\" (UniqueName: \"kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809697    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-trusted-ca\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809784    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809786    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8924cae-fc43-49bd-9bab-d4bc56220d6a-ovnkube-config\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809823    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.809846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809855    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-multus-certs\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809880    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dd41889-9433-4fdb-8e05-52ea68e32536-agent-certs\") pod \"konnectivity-agent-tnrf7\" (UID: \"3dd41889-9433-4fdb-8e05-52ea68e32536\") " pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809927    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809989    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-device-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.809994    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0e41a51-8711-47dd-a04d-af098a4d7fcb-trusted-ca\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810029    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9vz\" (UniqueName: \"kubernetes.io/projected/c0e41a51-8711-47dd-a04d-af098a4d7fcb-kube-api-access-vv9vz\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810049    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-kubelet\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810075    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810101    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810123    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-log-socket\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810150    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdptk\" (UniqueName: \"kubernetes.io/projected/f8924cae-fc43-49bd-9bab-d4bc56220d6a-kube-api-access-sdptk\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810173    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85738300-00ba-4ba6-be20-2b259922fd7d-host-slash\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810189    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-systemd-units\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810229    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810272    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810388    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810424    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-netd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810460    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f25010f1-45b5-4994-91fa-7de15f04fb08-original-pull-secret\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810466    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdlb7\" (UniqueName: \"kubernetes.io/projected/b60e34dd-a30c-4f35-9593-1935139f9a47-kube-api-access-fdlb7\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810499    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-tmp\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810523    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-multus\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810542    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bddd1e71-4116-437a-b7ee-873bc218fa0b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810560    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-var-lib-kubelet\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810568    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-tmp\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810579    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810612    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-daemon-config\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810680    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a04059dd-fc63-4523-829f-c0ce3c4b8129-config\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810690    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bddd1e71-4116-437a-b7ee-873bc218fa0b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810712    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-tuned\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810740    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cnibin\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810754    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-daemon-config\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810768    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-conf-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810797    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810804    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810808    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-tuned\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810846    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-bin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810870    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a04059dd-fc63-4523-829f-c0ce3c4b8129-config\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:16.811019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.810879    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr685\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.834627 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.834588    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4n2\" (UniqueName: \"kubernetes.io/projected/fba6806a-680a-4fd0-8d5e-2b02db024040-kube-api-access-kk4n2\") pod \"migrator-66df5d496b-s9s6q\" (UID: \"fba6806a-680a-4fd0-8d5e-2b02db024040\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s9s6q"
Oct 30 12:32:16.834814 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.834788    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvpv\" (UniqueName: \"kubernetes.io/projected/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-kube-api-access-vnvpv\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:32:16.834858 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.834808    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztc6\" (UniqueName: \"kubernetes.io/projected/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-kube-api-access-zztc6\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:16.834858 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.834802    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdsj7\" (UniqueName: \"kubernetes.io/projected/c5bb30ba-a903-4c97-80a8-6a1f398b6664-kube-api-access-pdsj7\") pod \"kas-connection-checker-685895c7c9-pbb9n\" (UID: \"c5bb30ba-a903-4c97-80a8-6a1f398b6664\") " pod="kube-system/kas-connection-checker-685895c7c9-pbb9n"
Oct 30 12:32:16.834858 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.834825    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7sd\" (UniqueName: \"kubernetes.io/projected/b02a9da2-ad91-4bac-8177-c851dc4a37d3-kube-api-access-rk7sd\") pod \"volume-data-source-validator-58c9f5f46d-4t2tb\" (UID: \"b02a9da2-ad91-4bac-8177-c851dc4a37d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4t2tb"
Oct 30 12:32:16.836147 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.836126    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjzd\" (UniqueName: \"kubernetes.io/projected/090c5691-fa68-4427-84ed-e7b4797f77c7-kube-api-access-gpjzd\") pod \"network-check-source-6f78d59f99-fc5ft\" (UID: \"090c5691-fa68-4427-84ed-e7b4797f77c7\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fc5ft"
Oct 30 12:32:16.838249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.838230    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ng7\" (UniqueName: \"kubernetes.io/projected/46fbee35-1719-4228-a543-874c0d3f4b4b-kube-api-access-k5ng7\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:32:16.838664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.838647    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rjf\" (UniqueName: \"kubernetes.io/projected/45f0654b-10d6-4a98-b6f9-7b679ac028a2-kube-api-access-j7rjf\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.840924 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.840906    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzq8v\" (UniqueName: \"kubernetes.io/projected/51f4f667-c671-472c-ae68-beb10d17b645-kube-api-access-kzq8v\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.845015 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.844991    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhms\" (UniqueName: \"kubernetes.io/projected/b97ac8d4-f132-4fa6-9f7f-d6d050098612-kube-api-access-qqhms\") pod \"migrator-66df5d496b-zl9k6\" (UID: \"b97ac8d4-f132-4fa6-9f7f-d6d050098612\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zl9k6"
Oct 30 12:32:16.845162 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.845129    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8j7n\" (UniqueName: \"kubernetes.io/projected/02e9f8c2-3da0-474f-b7db-40bee0e8d743-kube-api-access-d8j7n\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.845634 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.845616    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh7v\" (UniqueName: \"kubernetes.io/projected/693d053a-5b5c-4be2-9778-a4255c6f3417-kube-api-access-hgh7v\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:16.845909 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.845893    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.846156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.846140    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65894\" (UniqueName: \"kubernetes.io/projected/061ed4c7-604d-4cea-aedf-f7bbbb7a1c73-kube-api-access-65894\") pod \"network-check-target-fkhv4\" (UID: \"061ed4c7-604d-4cea-aedf-f7bbbb7a1c73\") " pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:16.848469 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.848450    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xknz\" (UniqueName: \"kubernetes.io/projected/0b3f7eca-7b44-46c3-b1fd-d9bb4f848539-kube-api-access-4xknz\") pod \"kas-connection-checker-685895c7c9-829vl\" (UID: \"0b3f7eca-7b44-46c3-b1fd-d9bb4f848539\") " pod="kube-system/kas-connection-checker-685895c7c9-829vl"
Oct 30 12:32:16.850059 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.850042    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlt79\" (UniqueName: \"kubernetes.io/projected/090d2edb-0fb8-4fa0-8d91-d866c06226b3-kube-api-access-dlt79\") pod \"kas-connection-checker-685895c7c9-vwdb5\" (UID: \"090d2edb-0fb8-4fa0-8d91-d866c06226b3\") " pod="kube-system/kas-connection-checker-685895c7c9-vwdb5"
Oct 30 12:32:16.850276 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.850259    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr685\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.855251 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.855140    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlhs\" (UniqueName: \"kubernetes.io/projected/85738300-00ba-4ba6-be20-2b259922fd7d-kube-api-access-tqlhs\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:32:16.855251 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.855155    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.855503 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.855479    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlgxt\" (UniqueName: \"kubernetes.io/projected/929ccbc1-e13f-4dba-b00f-11a9c96d62f8-kube-api-access-hlgxt\") pod \"insights-operator-7c4986d449-7tvl5\" (UID: \"929ccbc1-e13f-4dba-b00f-11a9c96d62f8\") " pod="openshift-insights/insights-operator-7c4986d449-7tvl5"
Oct 30 12:32:16.855596 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.855553    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn87g\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.855727 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.855666    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdjl\" (UniqueName: \"kubernetes.io/projected/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-kube-api-access-2wdjl\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.855961 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.855917    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrfb\" (UniqueName: \"kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb\") pod \"periodic-gathering-zks2p-9688t\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") " pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:32:16.856023 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.855992    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkrm\" (UniqueName: \"kubernetes.io/projected/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kube-api-access-vtkrm\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.856515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.856427    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9vz\" (UniqueName: \"kubernetes.io/projected/c0e41a51-8711-47dd-a04d-af098a4d7fcb-kube-api-access-vv9vz\") pod \"console-operator-7cd56b8dbc-lq5sc\" (UID: \"c0e41a51-8711-47dd-a04d-af098a4d7fcb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:16.856833 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.856727    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnb2q\" (UniqueName: \"kubernetes.io/projected/6662ac83-7fc2-451d-b26b-68d934badad1-kube-api-access-nnb2q\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:16.860024 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.860002    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9mw\" (UniqueName: \"kubernetes.io/projected/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-kube-api-access-8s9mw\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.862398 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.862373    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdptk\" (UniqueName: \"kubernetes.io/projected/f8924cae-fc43-49bd-9bab-d4bc56220d6a-kube-api-access-sdptk\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.863102 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.863075    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmxt\" (UniqueName: \"kubernetes.io/projected/179c5618-551b-4e6e-b0d3-5fa27708fb1f-kube-api-access-5tmxt\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.864464 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.864445    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsspd\" (UniqueName: \"kubernetes.io/projected/a04059dd-fc63-4523-829f-c0ce3c4b8129-kube-api-access-xsspd\") pod \"service-ca-operator-544d749c8c-rbk6k\" (UID: \"a04059dd-fc63-4523-829f-c0ce3c4b8129\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k"
Oct 30 12:32:16.864788 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.864766    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkwr\" (UniqueName: \"kubernetes.io/projected/bddd1e71-4116-437a-b7ee-873bc218fa0b-kube-api-access-cpkwr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs\" (UID: \"bddd1e71-4116-437a-b7ee-873bc218fa0b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs"
Oct 30 12:32:16.867171 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.867153    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdlb7\" (UniqueName: \"kubernetes.io/projected/b60e34dd-a30c-4f35-9593-1935139f9a47-kube-api-access-fdlb7\") pod \"ingress-canary-vd2n6\" (UID: \"b60e34dd-a30c-4f35-9593-1935139f9a47\") " pod="openshift-ingress-canary/ingress-canary-vd2n6"
Oct 30 12:32:16.911681 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911592    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cnibin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.911681 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911629    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-ovn\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.911681 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911650    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-system-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.911681 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911674    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-netns\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911715    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-cnibin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911715    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-ovn\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911744    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-system-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911779    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-netns\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911805    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-etc-kubernetes\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911824    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-sys-fs\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911841    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911859    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-hosts-file\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911882    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-conf\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911904    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-sys-fs\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.911918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911905    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911942    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4-hosts-file\") pod \"node-resolver-98w6d\" (UID: \"ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4\") " pod="openshift-dns/node-resolver-98w6d"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911916    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-etc-kubernetes\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911946    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.911984    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-host\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.911994    4993 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912033    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-conf\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912035    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912048    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysctl-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912055    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-host\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912084    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.412063731 +0000 UTC m=+1.887783320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912105    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-k8s-cni-cncf-io\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912148    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-netns\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912175    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-netns\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912175    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-k8s-cni-cncf-io\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912170    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.412157216 +0000 UTC m=+1.887876804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912256    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-run\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912290    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912323    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-modprobe-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912337    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912341    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-hostroot\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.912342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912335    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-run\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912365    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-dbus\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912383    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912401    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-kubelet\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912421    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912429    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-modprobe-d\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912442    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-socket-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912469    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-kubelet\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912477    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912471    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912473    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-dbus\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912474    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-hostroot\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912506    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-socket-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912511    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912525    4993 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912539    4993 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912561    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912570    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert podName:5d6d13bc-3169-4d2c-830c-66837db3faf4 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.412560726 +0000 UTC m=+1.888280319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-87dsg" (UID: "5d6d13bc-3169-4d2c-830c-66837db3faf4") : secret "networking-console-plugin-cert" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912588    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.412577713 +0000 UTC m=+1.888297303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912608    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.412594427 +0000 UTC m=+1.888314030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912640    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-var-lib-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912665    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-var-lib-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912665    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912694    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912707    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912709    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-multus-certs\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912717    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccff9c748-92chp: secret "image-registry-tls" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912724    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912740    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-device-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912755    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls podName:67405c0f-c406-4e55-ade6-ba7a26a99a38 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.41274413 +0000 UTC m=+1.888463757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls") pod "image-registry-6ccff9c748-92chp" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38") : secret "image-registry-tls" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912768    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-device-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912778    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-cni-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912781    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-kubelet\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912798    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-run-multus-certs\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912817    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-log-socket\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912835    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85738300-00ba-4ba6-be20-2b259922fd7d-host-slash\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912853    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-systemd-units\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912858    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912867    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-log-socket\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912879    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912890    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-kubelet\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912896    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85738300-00ba-4ba6-be20-2b259922fd7d-host-slash\") pod \"iptables-alerter-4mvs6\" (UID: \"85738300-00ba-4ba6-be20-2b259922fd7d\") " pod="openshift-network-operator/iptables-alerter-4mvs6"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912902    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-systemd-units\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912905    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.912918    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.412903408 +0000 UTC m=+1.888623010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : secret "metrics-daemon-secret" not found
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912934    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912951    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.912960    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-netd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913002    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-multus\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913030    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-var-lib-kubelet\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913030    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-netd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913059    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cnibin\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913072    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-multus\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913086    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-var-lib-kubelet\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913083    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-conf-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913116    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-conf-dir\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913121    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913134    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f78bb5-5e15-4574-93d6-9cdbdc2c95ba-cnibin\") pod \"multus-additional-cni-plugins-n6hfj\" (UID: \"53f78bb5-5e15-4574-93d6-9cdbdc2c95ba\") " pod="openshift-multus/multus-additional-cni-plugins-n6hfj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913149    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-bin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913155    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-etc-selinux\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913177    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-etc-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913201    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-host-var-lib-cni-bin\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913228    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-socket-dir-parent\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913233    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-etc-openvswitch\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913255    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913277    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysconfig\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.913243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913284    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e9f8c2-3da0-474f-b7db-40bee0e8d743-multus-socket-dir-parent\") pod \"multus-89m26\" (UID: \"02e9f8c2-3da0-474f-b7db-40bee0e8d743\") " pod="openshift-multus/multus-89m26"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913303    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-slash\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913323    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-sysconfig\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913323    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913336    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913370    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45f0654b-10d6-4a98-b6f9-7b679ac028a2-crio-socket\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913390    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-slash\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913397    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-systemd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913423    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-kubelet-config\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913427    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45f0654b-10d6-4a98-b6f9-7b679ac028a2-crio-socket\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913451    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913455    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-run-systemd\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.913457    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.413445837 +0000 UTC m=+1.889165425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913482    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f25010f1-45b5-4994-91fa-7de15f04fb08-kubelet-config\") pod \"global-pull-secret-syncer-s4gm9\" (UID: \"f25010f1-45b5-4994-91fa-7de15f04fb08\") " pod="kube-system/global-pull-secret-syncer-s4gm9"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913502    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913527    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-systemd\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913550    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-bin\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913560    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-systemd\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.913508    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.913591    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6676c6bb89-m66tl: secret "image-registry-tls" not found
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913575    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913608    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-cni-bin\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913622    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-kubernetes\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.913627    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls podName:fcbd57e7-45d9-472f-9afe-ded1285276b9 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.413616272 +0000 UTC m=+1.889335895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls") pod "image-registry-6676c6bb89-m66tl" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9") : secret "image-registry-tls" not found
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.913627    4993 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913657    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-etc-kubernetes\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:16.913662    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:17.413653951 +0000 UTC m=+1.889373579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : secret "router-metrics-certs-default" not found
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913654    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913656    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-registration-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913693    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-lib-modules\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913705    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37d4d8e7-29c1-425a-87e9-d7fd23dce294-registration-dir\") pod \"aws-ebs-csi-driver-node-d4tvp\" (UID: \"37d4d8e7-29c1-425a-87e9-d7fd23dce294\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d4tvp"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913711    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-node-log\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913737    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8924cae-fc43-49bd-9bab-d4bc56220d6a-node-log\") pod \"ovnkube-node-svlgw\" (UID: \"f8924cae-fc43-49bd-9bab-d4bc56220d6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913738    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-sys\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913770    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fbee35-1719-4228-a543-874c0d3f4b4b-host\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913775    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-sys\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913810    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6af58857-77e1-4f65-89e0-3f7fa2ea3a52-lib-modules\") pod \"tuned-sq7sj\" (UID: \"6af58857-77e1-4f65-89e0-3f7fa2ea3a52\") " pod="openshift-cluster-node-tuning-operator/tuned-sq7sj"
Oct 30 12:32:16.915528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.913844    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fbee35-1719-4228-a543-874c0d3f4b4b-host\") pod \"node-ca-wjpmw\" (UID: \"46fbee35-1719-4228-a543-874c0d3f4b4b\") " pod="openshift-image-registry/node-ca-wjpmw"
Oct 30 12:32:16.992394 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.992359    4993 scope.go:122] "RemoveContainer" containerID="5456cdf1ef89b8c177fbf99cbd62f12e9886cfd5ae61fdd92c2309066c57513a"
Oct 30 12:32:16.998516 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:16.998462    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-s9s6q_fba6806a-680a-4fd0-8d5e-2b02db024040/migrator/0.log"
Oct 30 12:32:17.215476 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.215395    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-zl9k6_b97ac8d4-f132-4fa6-9f7f-d6d050098612/migrator/0.log"
Oct 30 12:32:17.398972 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.398939    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs_bddd1e71-4116-437a-b7ee-873bc218fa0b/kube-storage-version-migrator-operator/0.log"
Oct 30 12:32:17.416947 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.416904    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.416965    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.416985    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.417022    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.417045    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.417063    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417061    4993 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417112    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417128    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:18.417115516 +0000 UTC m=+2.892835117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : secret "router-metrics-certs-default" not found
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417144    4993 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:17.417161 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417159    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:18.417145675 +0000 UTC m=+2.892865262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.417182    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417197    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:18.417181408 +0000 UTC m=+2.892901009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417149    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.417254    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.417300    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417312    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:18.417295779 +0000 UTC m=+2.893015387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417358    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:18.417350318 +0000 UTC m=+2.893069939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.417358    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417069    4993 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417441    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:18.417432519 +0000 UTC m=+2.893152141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417386    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.417476    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:18.417467215 +0000 UTC m=+2.893186815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : secret "metrics-daemon-secret" not found
Oct 30 12:32:17.417662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.417456    4993 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:32:17.419814 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.419793    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"image-registry-6ccff9c748-92chp\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") " pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:17.420012 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.419990    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"image-registry-6676c6bb89-m66tl\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") " pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:17.420051 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.419990    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d6d13bc-3169-4d2c-830c-66837db3faf4-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-87dsg\" (UID: \"5d6d13bc-3169-4d2c-830c-66837db3faf4\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:17.590981 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.590936    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:17.593231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.593112    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg"
Oct 30 12:32:17.603127 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.603090    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:17.755054 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.755023    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/2.log"
Oct 30 12:32:17.755574 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.755558    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/1.log"
Oct 30 12:32:17.755614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.755590    4993 generic.go:356] "Generic (PLEG): container finished" podID="c0e41a51-8711-47dd-a04d-af098a4d7fcb" containerID="e431845760ecc28e789661a085b58741d438b12b3dda92e8b98010bea3ea83bb" exitCode=255
Oct 30 12:32:17.755696 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.755671    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" event={"ID":"c0e41a51-8711-47dd-a04d-af098a4d7fcb","Type":"ContainerDied","Data":"e431845760ecc28e789661a085b58741d438b12b3dda92e8b98010bea3ea83bb"}
Oct 30 12:32:17.755749 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.755738    4993 scope.go:122] "RemoveContainer" containerID="5456cdf1ef89b8c177fbf99cbd62f12e9886cfd5ae61fdd92c2309066c57513a"
Oct 30 12:32:17.756220 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.756188    4993 scope.go:122] "RemoveContainer" containerID="e431845760ecc28e789661a085b58741d438b12b3dda92e8b98010bea3ea83bb"
Oct 30 12:32:17.756435 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:17.756407    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:17.908750 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.908667    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-87dsg"]
Oct 30 12:32:17.910893 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.910872    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6676c6bb89-m66tl"]
Oct 30 12:32:17.912669 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:17.912640    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6d13bc_3169_4d2c_830c_66837db3faf4.slice/crio-61b5ed4b3830b881d47999e8790e069676b4cd5ea1211ffbc355f13ebb355917 WatchSource:0}: Error finding container 61b5ed4b3830b881d47999e8790e069676b4cd5ea1211ffbc355f13ebb355917: Status 404 returned error can't find the container with id 61b5ed4b3830b881d47999e8790e069676b4cd5ea1211ffbc355f13ebb355917
Oct 30 12:32:17.915086 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.915067    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:32:17.915667 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:17.915642    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbd57e7_45d9_472f_9afe_ded1285276b9.slice/crio-fdb982d844189b554a83ef5df0864ff44eccd71f8e751d1ee2803dc6579cc7a8 WatchSource:0}: Error finding container fdb982d844189b554a83ef5df0864ff44eccd71f8e751d1ee2803dc6579cc7a8: Status 404 returned error can't find the container with id fdb982d844189b554a83ef5df0864ff44eccd71f8e751d1ee2803dc6579cc7a8
Oct 30 12:32:17.921943 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.921914    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ccff9c748-92chp"]
Oct 30 12:32:17.925666 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:17.925637    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67405c0f_c406_4e55_ade6_ba7a26a99a38.slice/crio-60aec1e1aa82591d328617d22c0e1866862dda7e1036ab8b406c2ea6a8d83922 WatchSource:0}: Error finding container 60aec1e1aa82591d328617d22c0e1866862dda7e1036ab8b406c2ea6a8d83922: Status 404 returned error can't find the container with id 60aec1e1aa82591d328617d22c0e1866862dda7e1036ab8b406c2ea6a8d83922
Oct 30 12:32:17.945039 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:17.945009    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:18.427127 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.427088    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.427135    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.427189    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427263    4993 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427302    4993 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427318    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427341    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.427313181 +0000 UTC m=+4.903032769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427360    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.427351701 +0000 UTC m=+4.903071288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427382    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.427365215 +0000 UTC m=+4.903084807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.427412    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:18.427468 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.427454    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:18.428000 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.427514    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:18.428000 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.427556    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:18.428000 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427520    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:18.428000 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427566    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:18.428000 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427622    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.427612268 +0000 UTC m=+4.903331899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:18.428000 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427680    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.427673394 +0000 UTC m=+4.903392983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:18.428000 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:18.427689    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:20.42768483 +0000 UTC m=+4.903404419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : secret "metrics-daemon-secret" not found
Oct 30 12:32:18.430420 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.430385    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f4f667-c671-472c-ae68-beb10d17b645-metrics-certs\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:18.764061 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.764024    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ccff9c748-92chp" event={"ID":"67405c0f-c406-4e55-ade6-ba7a26a99a38","Type":"ContainerStarted","Data":"30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c"}
Oct 30 12:32:18.764626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.764085    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ccff9c748-92chp" event={"ID":"67405c0f-c406-4e55-ade6-ba7a26a99a38","Type":"ContainerStarted","Data":"60aec1e1aa82591d328617d22c0e1866862dda7e1036ab8b406c2ea6a8d83922"}
Oct 30 12:32:18.764626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.764130    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:18.765903 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.765880    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/2.log"
Oct 30 12:32:18.767348 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.767309    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg" event={"ID":"5d6d13bc-3169-4d2c-830c-66837db3faf4","Type":"ContainerStarted","Data":"61b5ed4b3830b881d47999e8790e069676b4cd5ea1211ffbc355f13ebb355917"}
Oct 30 12:32:18.768919 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.768881    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl" event={"ID":"fcbd57e7-45d9-472f-9afe-ded1285276b9","Type":"ContainerStarted","Data":"511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932"}
Oct 30 12:32:18.769053 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:18.768930    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl" event={"ID":"fcbd57e7-45d9-472f-9afe-ded1285276b9","Type":"ContainerStarted","Data":"fdb982d844189b554a83ef5df0864ff44eccd71f8e751d1ee2803dc6579cc7a8"}
Oct 30 12:32:19.775156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:19.775113    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-87dsg" event={"ID":"5d6d13bc-3169-4d2c-830c-66837db3faf4","Type":"ContainerStarted","Data":"8f6316c1aeb706e0c5334e5d4b1545a0abf144bfe6d08e23ea849841aa07d46a"}
Oct 30 12:32:19.775693 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:19.775225    4993 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:32:20.071418 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:20.071324    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:20.149828 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:20.149174    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:20.449579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:20.449485    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:20.449579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:20.449528    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:20.449579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:20.449565    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:20.449589    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:20.449610    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:20.449643    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449652    4993 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449713    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449727    4993 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449736    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls podName:fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.449716291 +0000 UTC m=+8.925435902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8pxzl" (UID: "fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20") : secret "cluster-monitoring-operator-tls" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449755    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.449748631 +0000 UTC m=+8.925468219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449721    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449767    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.449761223 +0000 UTC m=+8.925480811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : secret "metrics-daemon-secret" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449777    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls podName:693d053a-5b5c-4be2-9778-a4255c6f3417 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.449772525 +0000 UTC m=+8.925492113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-x56vx" (UID: "693d053a-5b5c-4be2-9778-a4255c6f3417") : secret "samples-operator-tls" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449651    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449790    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.449783139 +0000 UTC m=+8.925502727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:20.449838 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:20.449807    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls podName:179c5618-551b-4e6e-b0d3-5fa27708fb1f nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.449795677 +0000 UTC m=+8.925515265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls") pod "dns-default-qngfl" (UID: "179c5618-551b-4e6e-b0d3-5fa27708fb1f") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:22.395895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:22.395849    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:22.396352 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:22.396257    4993 scope.go:122] "RemoveContainer" containerID="e431845760ecc28e789661a085b58741d438b12b3dda92e8b98010bea3ea83bb"
Oct 30 12:32:22.396463 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:22.396442    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:23.804399 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:23.804355    4993 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:23.804810 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:23.804749    4993 scope.go:122] "RemoveContainer" containerID="e431845760ecc28e789661a085b58741d438b12b3dda92e8b98010bea3ea83bb"
Oct 30 12:32:23.804955 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:23.804935    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:24.489308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.489251    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:24.489308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.489311    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:24.489601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.489332    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:24.489601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.489373    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:24.489601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.489414    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:24.489601 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:24.489427    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:24.489601 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:24.489498    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls podName:45f0654b-10d6-4a98-b6f9-7b679ac028a2 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:32.489483063 +0000 UTC m=+16.965202652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gl9mb" (UID: "45f0654b-10d6-4a98-b6f9-7b679ac028a2") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:24.489601 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:24.489508    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:24.489601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.489438    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:24.489601 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:24.489603    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs podName:6662ac83-7fc2-451d-b26b-68d934badad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:32.489584974 +0000 UTC m=+16.965304572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs") pod "network-metrics-daemon-7dxkx" (UID: "6662ac83-7fc2-451d-b26b-68d934badad1") : secret "metrics-daemon-secret" not found
Oct 30 12:32:24.489929 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:24.489624    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle podName:51f4f667-c671-472c-ae68-beb10d17b645 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:32.489614605 +0000 UTC m=+16.965334198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle") pod "router-default-77bb4759d7-ztvnj" (UID: "51f4f667-c671-472c-ae68-beb10d17b645") : configmap references non-existent config key: service-ca.crt
Oct 30 12:32:24.491871 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.491838    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/179c5618-551b-4e6e-b0d3-5fa27708fb1f-metrics-tls\") pod \"dns-default-qngfl\" (UID: \"179c5618-551b-4e6e-b0d3-5fa27708fb1f\") " pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:24.492024 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.492000    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8pxzl\" (UID: \"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:24.492104 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.492079    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/693d053a-5b5c-4be2-9778-a4255c6f3417-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-x56vx\" (UID: \"693d053a-5b5c-4be2-9778-a4255c6f3417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:24.503987 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.503950    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:24.789660 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.789564    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"
Oct 30 12:32:24.790583 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.790562    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"
Oct 30 12:32:24.797854 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:24.797827    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qngfl"]
Oct 30 12:32:25.094168 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:25.094086    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx"]
Oct 30 12:32:25.103448 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:25.103412    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl"]
Oct 30 12:32:25.110425 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:25.110390    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1c2c87_7cd8_4a13_a4a0_ff95ddffdd20.slice/crio-2af2cb1429796b1a225e6fd8be1c5830ac8a9b1e5fff9b3f5d6cad534a5baeb6 WatchSource:0}: Error finding container 2af2cb1429796b1a225e6fd8be1c5830ac8a9b1e5fff9b3f5d6cad534a5baeb6: Status 404 returned error can't find the container with id 2af2cb1429796b1a225e6fd8be1c5830ac8a9b1e5fff9b3f5d6cad534a5baeb6
Oct 30 12:32:25.760958 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:25.759379    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:25.760958 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:25.759622    4993 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:32:25.802258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:25.802194    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl" event={"ID":"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20","Type":"ContainerStarted","Data":"2af2cb1429796b1a225e6fd8be1c5830ac8a9b1e5fff9b3f5d6cad534a5baeb6"}
Oct 30 12:32:25.804084 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:25.804049    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qngfl" event={"ID":"179c5618-551b-4e6e-b0d3-5fa27708fb1f","Type":"ContainerStarted","Data":"6721191465c4b5c634ec1d7811ff4e3f0c4f61ae3d44cf1081139b52df92a4d7"}
Oct 30 12:32:25.805863 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:25.805832    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx" event={"ID":"693d053a-5b5c-4be2-9778-a4255c6f3417","Type":"ContainerStarted","Data":"ace6a0332851c2c35e5d4ea0319ebae41a9a7a3a4e652dc529ad9f584a64adac"}
Oct 30 12:32:26.817299 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:26.817259    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qngfl" event={"ID":"179c5618-551b-4e6e-b0d3-5fa27708fb1f","Type":"ContainerStarted","Data":"1039a50f131618339ff28699afa161af846fb1a1844eca3021b5267788123768"}
Oct 30 12:32:27.684129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.684090    4993 scope.go:122] "RemoveContainer" containerID="e431845760ecc28e789661a085b58741d438b12b3dda92e8b98010bea3ea83bb"
Oct 30 12:32:27.823796 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.823758    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qngfl" event={"ID":"179c5618-551b-4e6e-b0d3-5fa27708fb1f","Type":"ContainerStarted","Data":"2e8c7355d783b436184d1d79a32dbbcc1f91bac54ad149494700c3d422283aff"}
Oct 30 12:32:27.824295 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.823839    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:27.826582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.826545    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx" event={"ID":"693d053a-5b5c-4be2-9778-a4255c6f3417","Type":"ContainerStarted","Data":"2d0cece1887075ff30bc8299491f7ba2f32ec00af05bea0e1eba9ef0bcbcd6b5"}
Oct 30 12:32:27.826747 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.826612    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-x56vx" event={"ID":"693d053a-5b5c-4be2-9778-a4255c6f3417","Type":"ContainerStarted","Data":"811c19c7cb3251bc79ccaa36a696f2739d076cd541837bf847d0900a4b4edad4"}
Oct 30 12:32:27.828395 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.828366    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8pxzl" event={"ID":"fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20","Type":"ContainerStarted","Data":"46a81c328fc254eded181a12ef76fcfcb556645b52dbc2d0884dc8a4be23cb83"}
Oct 30 12:32:27.829999 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.829979    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:32:27.830402 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.830385    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/2.log"
Oct 30 12:32:27.830473 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.830418    4993 generic.go:356] "Generic (PLEG): container finished" podID="c0e41a51-8711-47dd-a04d-af098a4d7fcb" containerID="d953175ce339b3a0cf8a355a4144741fe29e706fe019504e68e02b5e17063a4d" exitCode=255
Oct 30 12:32:27.830473 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.830443    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" event={"ID":"c0e41a51-8711-47dd-a04d-af098a4d7fcb","Type":"ContainerDied","Data":"d953175ce339b3a0cf8a355a4144741fe29e706fe019504e68e02b5e17063a4d"}
Oct 30 12:32:27.830548 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.830480    4993 scope.go:122] "RemoveContainer" containerID="e431845760ecc28e789661a085b58741d438b12b3dda92e8b98010bea3ea83bb"
Oct 30 12:32:27.830779 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:27.830760    4993 scope.go:122] "RemoveContainer" containerID="d953175ce339b3a0cf8a355a4144741fe29e706fe019504e68e02b5e17063a4d"
Oct 30 12:32:27.830988 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:27.830966    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:28.835844 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:28.835814    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:32:31.068900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:31.068865    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6676c6bb89-m66tl"]
Oct 30 12:32:32.396200 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.396158    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:32.396691 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.396557    4993 scope.go:122] "RemoveContainer" containerID="d953175ce339b3a0cf8a355a4144741fe29e706fe019504e68e02b5e17063a4d"
Oct 30 12:32:32.396772 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:32.396753    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:32.558023 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.557984    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:32.558179 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.558041    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:32.558179 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.558104    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:32.558735 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.558707    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51f4f667-c671-472c-ae68-beb10d17b645-service-ca-bundle\") pod \"router-default-77bb4759d7-ztvnj\" (UID: \"51f4f667-c671-472c-ae68-beb10d17b645\") " pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:32.560461 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.560437    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6662ac83-7fc2-451d-b26b-68d934badad1-metrics-certs\") pod \"network-metrics-daemon-7dxkx\" (UID: \"6662ac83-7fc2-451d-b26b-68d934badad1\") " pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:32.560461 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.560445    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45f0654b-10d6-4a98-b6f9-7b679ac028a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gl9mb\" (UID: \"45f0654b-10d6-4a98-b6f9-7b679ac028a2\") " pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:32.592484 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.592441    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:32.601221 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.601179    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7dxkx"
Oct 30 12:32:32.605235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.605193    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gl9mb"
Oct 30 12:32:32.919672 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.919597    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7dxkx"]
Oct 30 12:32:32.922838 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:32.922803    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6662ac83_7fc2_451d_b26b_68d934badad1.slice/crio-10f29f0118a8d04e1f64ce17ccd427dee03be064ad8d497bc957efa3f48d6115 WatchSource:0}: Error finding container 10f29f0118a8d04e1f64ce17ccd427dee03be064ad8d497bc957efa3f48d6115: Status 404 returned error can't find the container with id 10f29f0118a8d04e1f64ce17ccd427dee03be064ad8d497bc957efa3f48d6115
Oct 30 12:32:32.928962 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:32.928930    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f0654b_10d6_4a98_b6f9_7b679ac028a2.slice/crio-7fed19be677c2b91b3a1dcae7ca3ed2bf40696ca2d027e423e4ef6f2d8194506 WatchSource:0}: Error finding container 7fed19be677c2b91b3a1dcae7ca3ed2bf40696ca2d027e423e4ef6f2d8194506: Status 404 returned error can't find the container with id 7fed19be677c2b91b3a1dcae7ca3ed2bf40696ca2d027e423e4ef6f2d8194506
Oct 30 12:32:32.931779 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.931706    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gl9mb"]
Oct 30 12:32:32.933688 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:32.933664    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f4f667_c671_472c_ae68_beb10d17b645.slice/crio-e0870fc93b71be22ccb787087e0f1f92a73507a57de5c85ee06442e8c6e3b3cc WatchSource:0}: Error finding container e0870fc93b71be22ccb787087e0f1f92a73507a57de5c85ee06442e8c6e3b3cc: Status 404 returned error can't find the container with id e0870fc93b71be22ccb787087e0f1f92a73507a57de5c85ee06442e8c6e3b3cc
Oct 30 12:32:32.938509 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:32.938485    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77bb4759d7-ztvnj"]
Oct 30 12:32:33.393122 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.393089    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tnrf7"
Oct 30 12:32:33.805080 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.805037    4993 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:33.805638 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.805616    4993 scope.go:122] "RemoveContainer" containerID="d953175ce339b3a0cf8a355a4144741fe29e706fe019504e68e02b5e17063a4d"
Oct 30 12:32:33.805912 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:33.805884    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-lq5sc_openshift-console-operator(c0e41a51-8711-47dd-a04d-af098a4d7fcb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" podUID="c0e41a51-8711-47dd-a04d-af098a4d7fcb"
Oct 30 12:32:33.863591 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.863555    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gl9mb" event={"ID":"45f0654b-10d6-4a98-b6f9-7b679ac028a2","Type":"ContainerStarted","Data":"62696c08d96d19450c002722958aee9ada137d5ce3ae27ee5f23da549dfe148e"}
Oct 30 12:32:33.863729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.863651    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gl9mb" event={"ID":"45f0654b-10d6-4a98-b6f9-7b679ac028a2","Type":"ContainerStarted","Data":"42d4093ad810dc70acffe5be5f5ecf24848336a2b71ed21b5ac7ee019021043b"}
Oct 30 12:32:33.863729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.863671    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gl9mb" event={"ID":"45f0654b-10d6-4a98-b6f9-7b679ac028a2","Type":"ContainerStarted","Data":"7fed19be677c2b91b3a1dcae7ca3ed2bf40696ca2d027e423e4ef6f2d8194506"}
Oct 30 12:32:33.865469 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.865438    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7dxkx" event={"ID":"6662ac83-7fc2-451d-b26b-68d934badad1","Type":"ContainerStarted","Data":"b18b7120d719c02aeb1b7f706ea2d687e7b382847bccfaba48f91ed1c42a2ee2"}
Oct 30 12:32:33.865573 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.865497    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7dxkx" event={"ID":"6662ac83-7fc2-451d-b26b-68d934badad1","Type":"ContainerStarted","Data":"10f29f0118a8d04e1f64ce17ccd427dee03be064ad8d497bc957efa3f48d6115"}
Oct 30 12:32:33.867012 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.866984    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77bb4759d7-ztvnj" event={"ID":"51f4f667-c671-472c-ae68-beb10d17b645","Type":"ContainerStarted","Data":"69d0ef100ddc345649d4453a07fdd950354a0e655d4df6f9afe9a81aa13fc105"}
Oct 30 12:32:33.867120 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:33.867033    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77bb4759d7-ztvnj" event={"ID":"51f4f667-c671-472c-ae68-beb10d17b645","Type":"ContainerStarted","Data":"e0870fc93b71be22ccb787087e0f1f92a73507a57de5c85ee06442e8c6e3b3cc"}
Oct 30 12:32:34.593084 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:34.593041    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:34.596075 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:34.596031    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:34.873276 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:34.873148    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7dxkx" event={"ID":"6662ac83-7fc2-451d-b26b-68d934badad1","Type":"ContainerStarted","Data":"ccd98d5c88d740252a28f7de4580755e514c74865c7aef2cdd0190b5d03197a0"}
Oct 30 12:32:34.875535 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:34.875497    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gl9mb" event={"ID":"45f0654b-10d6-4a98-b6f9-7b679ac028a2","Type":"ContainerStarted","Data":"854f33c5cdcb9db7a2e2c7e5461b17f01a1db81985d8577aef86cf027b77e25f"}
Oct 30 12:32:34.875690 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:34.875672    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:34.877041 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:34.877018    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77bb4759d7-ztvnj"
Oct 30 12:32:36.589177 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:36.589139    4993 patch_prober.go:29] interesting pod/image-registry-6ccff9c748-92chp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 12:32:36.589597 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:36.589195    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6ccff9c748-92chp" podUID="67405c0f-c406-4e55-ade6-ba7a26a99a38" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:32:37.838573 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:37.838540    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qngfl"
Oct 30 12:32:38.568890 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:38.568851    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:32:38.926244 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:38.926148    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4pqg2"]
Oct 30 12:32:38.948451 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:38.948408    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.004771 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.004724    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:32:39.004771 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.004758    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:32:39.004771 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.004774    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f8qlc\""
Oct 30 12:32:39.005064 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.004768    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:32:39.005064 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.004839    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:32:39.011315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011278    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-root\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.011497 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011321    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-wtmp\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.011497 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011340    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e97fe730-6f69-4c33-b12f-280022e5303b-metrics-client-ca\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.011497 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011425    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-sys\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.011497 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011464    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-textfile\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.011497 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011489    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.011763 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011527    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.011763 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011653    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-tls\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.011763 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.011689    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8rg\" (UniqueName: \"kubernetes.io/projected/e97fe730-6f69-4c33-b12f-280022e5303b-kube-api-access-rj8rg\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112535    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-tls\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112594    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8rg\" (UniqueName: \"kubernetes.io/projected/e97fe730-6f69-4c33-b12f-280022e5303b-kube-api-access-rj8rg\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112621    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-root\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112640    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-wtmp\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112664    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e97fe730-6f69-4c33-b12f-280022e5303b-metrics-client-ca\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112684    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-sys\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112727    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-sys\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112731    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-root\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112792    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-textfile\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112818    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-wtmp\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.112874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112822    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.113396 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.112894    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.113396 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.113238    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-textfile\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.113396 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.113382    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e97fe730-6f69-4c33-b12f-280022e5303b-metrics-client-ca\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.113396 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.113391    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.115175 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.115152    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.115304 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.115262    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e97fe730-6f69-4c33-b12f-280022e5303b-node-exporter-tls\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.135630 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.135598    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8rg\" (UniqueName: \"kubernetes.io/projected/e97fe730-6f69-4c33-b12f-280022e5303b-kube-api-access-rj8rg\") pod \"node-exporter-4pqg2\" (UID: \"e97fe730-6f69-4c33-b12f-280022e5303b\") " pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.284986 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.284939    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4pqg2"
Oct 30 12:32:39.335012 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:32:39.334978    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97fe730_6f69_4c33_b12f_280022e5303b.slice/crio-f3b5bc4b6d7c0d3524ed38d21eec2099acf20909ddf3f2f5500f9a4ef69e4cdb WatchSource:0}: Error finding container f3b5bc4b6d7c0d3524ed38d21eec2099acf20909ddf3f2f5500f9a4ef69e4cdb: Status 404 returned error can't find the container with id f3b5bc4b6d7c0d3524ed38d21eec2099acf20909ddf3f2f5500f9a4ef69e4cdb
Oct 30 12:32:39.779441 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.779411    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:32:39.905237 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:39.899662    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pqg2" event={"ID":"e97fe730-6f69-4c33-b12f-280022e5303b","Type":"ContainerStarted","Data":"f3b5bc4b6d7c0d3524ed38d21eec2099acf20909ddf3f2f5500f9a4ef69e4cdb"}
Oct 30 12:32:40.905421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:40.905326    4993 generic.go:356] "Generic (PLEG): container finished" podID="e97fe730-6f69-4c33-b12f-280022e5303b" containerID="ad4d6a7e5d4cd326087c6bdb46d106532c7fc305c05360d5e13224e538a23942" exitCode=0
Oct 30 12:32:40.905790 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:40.905415    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pqg2" event={"ID":"e97fe730-6f69-4c33-b12f-280022e5303b","Type":"ContainerDied","Data":"ad4d6a7e5d4cd326087c6bdb46d106532c7fc305c05360d5e13224e538a23942"}
Oct 30 12:32:41.075302 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:41.075278    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:41.575769 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:41.575729    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-svlgw"
Oct 30 12:32:41.912103 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:41.912016    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pqg2" event={"ID":"e97fe730-6f69-4c33-b12f-280022e5303b","Type":"ContainerStarted","Data":"4a8462387de19323df98c9c21ca0162b1c4cf3526b4747e82a4ee02f6e463e6a"}
Oct 30 12:32:41.912103 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:41.912077    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pqg2" event={"ID":"e97fe730-6f69-4c33-b12f-280022e5303b","Type":"ContainerStarted","Data":"7ef1d25c15a564c91f47d593282c88bfbaaf50c7fddc4d25328e7f03902c5027"}
Oct 30 12:32:41.955527 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:41.955465    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4pqg2" podStartSLOduration=2.573470966 podStartE2EDuration="3.955309854s" totalImagesPullingTime="1.293194806s" totalInitContainerRuntime="88.644082ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:41.954893317 +0000 UTC m=+26.430612927" watchObservedRunningTime="2026-10-30 12:32:41.955309854 +0000 UTC m=+26.431029465"
Oct 30 12:32:47.948124 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:47.948091    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fkhv4"
Oct 30 12:32:54.684626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:54.684592    4993 scope.go:122] "RemoveContainer" containerID="d953175ce339b3a0cf8a355a4144741fe29e706fe019504e68e02b5e17063a4d"
Oct 30 12:32:54.965716 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:54.965639    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:32:54.965958 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:54.965770    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc" event={"ID":"c0e41a51-8711-47dd-a04d-af098a4d7fcb","Type":"ContainerStarted","Data":"640336ad4496f5c612a91018dbce2e5761498d0598c157e888bae09a3d17e17a"}
Oct 30 12:32:54.966068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:54.966051    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:55.268354 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:55.268326    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-lq5sc"
Oct 30 12:32:56.051165 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.051121    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ccff9c748-92chp"]
Oct 30 12:32:56.089003 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.088950    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl" podUID="fcbd57e7-45d9-472f-9afe-ded1285276b9" containerName="registry" containerID="cri-o://511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932" gracePeriod=30
Oct 30 12:32:56.336412 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.336384    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:56.467986 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.467948    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets\") pod \"fcbd57e7-45d9-472f-9afe-ded1285276b9\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") "
Oct 30 12:32:56.467986 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.467992    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration\") pod \"fcbd57e7-45d9-472f-9afe-ded1285276b9\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") "
Oct 30 12:32:56.468199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.468057    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685\") pod \"fcbd57e7-45d9-472f-9afe-ded1285276b9\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") "
Oct 30 12:32:56.468199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.468085    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates\") pod \"fcbd57e7-45d9-472f-9afe-ded1285276b9\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") "
Oct 30 12:32:56.468199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.468118    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted\") pod \"fcbd57e7-45d9-472f-9afe-ded1285276b9\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") "
Oct 30 12:32:56.468199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.468160    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca\") pod \"fcbd57e7-45d9-472f-9afe-ded1285276b9\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") "
Oct 30 12:32:56.468199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.468188    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token\") pod \"fcbd57e7-45d9-472f-9afe-ded1285276b9\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") "
Oct 30 12:32:56.468483 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.468268    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") pod \"fcbd57e7-45d9-472f-9afe-ded1285276b9\" (UID: \"fcbd57e7-45d9-472f-9afe-ded1285276b9\") "
Oct 30 12:32:56.470162 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.468776    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates" pod "fcbd57e7-45d9-472f-9afe-ded1285276b9" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:32:56.470162 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.468845    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca" pod "fcbd57e7-45d9-472f-9afe-ded1285276b9" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:32:56.470718 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.470680    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token" pod "fcbd57e7-45d9-472f-9afe-ded1285276b9" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:56.470808 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.470772    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration" pod "fcbd57e7-45d9-472f-9afe-ded1285276b9" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:32:56.470952 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.470928    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets" pod "fcbd57e7-45d9-472f-9afe-ded1285276b9" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:32:56.471035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.470935    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685" pod "fcbd57e7-45d9-472f-9afe-ded1285276b9" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9"). InnerVolumeSpecName "kube-api-access-kr685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:56.471035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.470992    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls" pod "fcbd57e7-45d9-472f-9afe-ded1285276b9" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:56.475907 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.475864    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted" pod "fcbd57e7-45d9-472f-9afe-ded1285276b9" (UID: "fcbd57e7-45d9-472f-9afe-ded1285276b9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:56.569421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.569336    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr685\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-kube-api-access-kr685\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:32:56.569421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.569368    4993 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-certificates\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:32:56.569421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.569378    4993 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcbd57e7-45d9-472f-9afe-ded1285276b9-ca-trust-extracted\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:32:56.569421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.569389    4993 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcbd57e7-45d9-472f-9afe-ded1285276b9-trusted-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:32:56.569421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.569397    4993 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-bound-sa-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:32:56.569421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.569406    4993 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcbd57e7-45d9-472f-9afe-ded1285276b9-registry-tls\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:32:56.569421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.569414    4993 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-installation-pull-secrets\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:32:56.569421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.569424    4993 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcbd57e7-45d9-472f-9afe-ded1285276b9-image-registry-private-configuration\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:32:56.978632 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.978601    4993 generic.go:356] "Generic (PLEG): container finished" podID="fcbd57e7-45d9-472f-9afe-ded1285276b9" containerID="511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932" exitCode=0
Oct 30 12:32:56.978815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.978651    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl"
Oct 30 12:32:56.978815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.978695    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl" event={"ID":"fcbd57e7-45d9-472f-9afe-ded1285276b9","Type":"ContainerDied","Data":"511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932"}
Oct 30 12:32:56.978815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.978750    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6676c6bb89-m66tl" event={"ID":"fcbd57e7-45d9-472f-9afe-ded1285276b9","Type":"ContainerDied","Data":"fdb982d844189b554a83ef5df0864ff44eccd71f8e751d1ee2803dc6579cc7a8"}
Oct 30 12:32:56.978815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.978783    4993 scope.go:122] "RemoveContainer" containerID="511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932"
Oct 30 12:32:56.987794 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.987774    4993 scope.go:122] "RemoveContainer" containerID="511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932"
Oct 30 12:32:56.988140 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:32:56.988113    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932\": container with ID starting with 511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932 not found: ID does not exist" containerID="511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932"
Oct 30 12:32:56.988272 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:56.988150    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932"} err="failed to get container status \"511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932\": rpc error: code = NotFound desc = could not find container \"511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932\": container with ID starting with 511a5d5a5e24507616c368cfa77f0a07382b39fdc13e4e6dae71354b2818a932 not found: ID does not exist"
Oct 30 12:32:57.035058 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:57.035018    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6676c6bb89-m66tl"]
Oct 30 12:32:57.047222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:57.047181    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6676c6bb89-m66tl"]
Oct 30 12:32:57.689869 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:32:57.689792    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcbd57e7-45d9-472f-9afe-ded1285276b9" path="/var/lib/kubelet/pods/fcbd57e7-45d9-472f-9afe-ded1285276b9/volumes"
Oct 30 12:33:08.569074 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:33:08.569033    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:33:21.072835 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.072785    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6ccff9c748-92chp" podUID="67405c0f-c406-4e55-ade6-ba7a26a99a38" containerName="registry" containerID="cri-o://30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c" gracePeriod=30
Oct 30 12:33:21.330700 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.330619    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:33:21.393760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.393723    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates\") pod \"67405c0f-c406-4e55-ade6-ba7a26a99a38\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") "
Oct 30 12:33:21.393760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.393764    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") pod \"67405c0f-c406-4e55-ade6-ba7a26a99a38\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") "
Oct 30 12:33:21.394014 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.393788    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca\") pod \"67405c0f-c406-4e55-ade6-ba7a26a99a38\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") "
Oct 30 12:33:21.394014 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.393817    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g\") pod \"67405c0f-c406-4e55-ade6-ba7a26a99a38\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") "
Oct 30 12:33:21.394014 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.393854    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets\") pod \"67405c0f-c406-4e55-ade6-ba7a26a99a38\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") "
Oct 30 12:33:21.394248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.394219    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca" pod "67405c0f-c406-4e55-ade6-ba7a26a99a38" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:21.394394 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.394372    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates" pod "67405c0f-c406-4e55-ade6-ba7a26a99a38" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:21.396172 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.396144    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls" pod "67405c0f-c406-4e55-ade6-ba7a26a99a38" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:21.396172 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.396162    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets" pod "67405c0f-c406-4e55-ade6-ba7a26a99a38" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:21.396326 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.396150    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g" pod "67405c0f-c406-4e55-ade6-ba7a26a99a38" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38"). InnerVolumeSpecName "kube-api-access-fn87g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:21.494479 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.494441    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token\") pod \"67405c0f-c406-4e55-ade6-ba7a26a99a38\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") "
Oct 30 12:33:21.494659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.494490    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration\") pod \"67405c0f-c406-4e55-ade6-ba7a26a99a38\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") "
Oct 30 12:33:21.494659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.494520    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted\") pod \"67405c0f-c406-4e55-ade6-ba7a26a99a38\" (UID: \"67405c0f-c406-4e55-ade6-ba7a26a99a38\") "
Oct 30 12:33:21.494659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.494629    4993 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-certificates\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:21.494659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.494641    4993 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-registry-tls\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:21.494659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.494650    4993 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67405c0f-c406-4e55-ade6-ba7a26a99a38-trusted-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:21.494659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.494659    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn87g\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-kube-api-access-fn87g\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:21.494873 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.494668    4993 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-installation-pull-secrets\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:21.496695 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.496663    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token" pod "67405c0f-c406-4e55-ade6-ba7a26a99a38" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:21.496695 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.496680    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration" pod "67405c0f-c406-4e55-ade6-ba7a26a99a38" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:21.501078 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.501044    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted" pod "67405c0f-c406-4e55-ade6-ba7a26a99a38" (UID: "67405c0f-c406-4e55-ade6-ba7a26a99a38"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:33:21.595830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.595734    4993 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67405c0f-c406-4e55-ade6-ba7a26a99a38-bound-sa-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:21.595830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.595785    4993 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67405c0f-c406-4e55-ade6-ba7a26a99a38-image-registry-private-configuration\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:21.595830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:21.595796    4993 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67405c0f-c406-4e55-ade6-ba7a26a99a38-ca-trust-extracted\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:22.078379 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.078343    4993 generic.go:356] "Generic (PLEG): container finished" podID="67405c0f-c406-4e55-ade6-ba7a26a99a38" containerID="30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c" exitCode=0
Oct 30 12:33:22.078817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.078379    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ccff9c748-92chp" event={"ID":"67405c0f-c406-4e55-ade6-ba7a26a99a38","Type":"ContainerDied","Data":"30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c"}
Oct 30 12:33:22.078817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.078421    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccff9c748-92chp"
Oct 30 12:33:22.078817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.078464    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ccff9c748-92chp" event={"ID":"67405c0f-c406-4e55-ade6-ba7a26a99a38","Type":"ContainerDied","Data":"60aec1e1aa82591d328617d22c0e1866862dda7e1036ab8b406c2ea6a8d83922"}
Oct 30 12:33:22.078817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.078493    4993 scope.go:122] "RemoveContainer" containerID="30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c"
Oct 30 12:33:22.093286 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.093258    4993 scope.go:122] "RemoveContainer" containerID="30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c"
Oct 30 12:33:22.093661 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:33:22.093593    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c\": container with ID starting with 30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c not found: ID does not exist" containerID="30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c"
Oct 30 12:33:22.093711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.093673    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c"} err="failed to get container status \"30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c\": rpc error: code = NotFound desc = could not find container \"30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c\": container with ID starting with 30eb9348255644b21bd4675b5ed3f92fabfd32b7d763f90b4113153fbb42494c not found: ID does not exist"
Oct 30 12:33:22.144108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.144070    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ccff9c748-92chp"]
Oct 30 12:33:22.156167 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:22.156130    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6ccff9c748-92chp"]
Oct 30 12:33:23.689320 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:23.689288    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67405c0f-c406-4e55-ade6-ba7a26a99a38" path="/var/lib/kubelet/pods/67405c0f-c406-4e55-ade6-ba7a26a99a38/volumes"
Oct 30 12:33:26.097321 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.097289    4993 generic.go:356] "Generic (PLEG): container finished" podID="a04059dd-fc63-4523-829f-c0ce3c4b8129" containerID="5f2890d739f5ee46a85ad9b5640f4255d4b85dd362fbdf83473a1b70735936ba" exitCode=0
Oct 30 12:33:26.097770 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.097364    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k" event={"ID":"a04059dd-fc63-4523-829f-c0ce3c4b8129","Type":"ContainerDied","Data":"5f2890d739f5ee46a85ad9b5640f4255d4b85dd362fbdf83473a1b70735936ba"}
Oct 30 12:33:26.097770 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.097723    4993 scope.go:122] "RemoveContainer" containerID="5f2890d739f5ee46a85ad9b5640f4255d4b85dd362fbdf83473a1b70735936ba"
Oct 30 12:33:26.098998 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.098972    4993 generic.go:356] "Generic (PLEG): container finished" podID="929ccbc1-e13f-4dba-b00f-11a9c96d62f8" containerID="9b57ee02edb0d385d45b0add4474a998e0bcfb33de540bec3d02762d277c34e9" exitCode=0
Oct 30 12:33:26.099103 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.099041    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7tvl5" event={"ID":"929ccbc1-e13f-4dba-b00f-11a9c96d62f8","Type":"ContainerDied","Data":"9b57ee02edb0d385d45b0add4474a998e0bcfb33de540bec3d02762d277c34e9"}
Oct 30 12:33:26.099359 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.099343    4993 scope.go:122] "RemoveContainer" containerID="9b57ee02edb0d385d45b0add4474a998e0bcfb33de540bec3d02762d277c34e9"
Oct 30 12:33:26.546222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.546176    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-zks2p-9688t"]
Oct 30 12:33:26.546455 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.546410    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-zks2p-9688t" podUID="8c2ec818-97ad-4634-9c26-d1f6803d439d" containerName="insights-gathering" containerID="cri-o://fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da" gracePeriod=30
Oct 30 12:33:26.712403 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.712357    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tnq7c-6db6s"]
Oct 30 12:33:26.712734 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.712718    4993 cpu_manager.go:414] "containerMap: removing container" podUID="67405c0f-c406-4e55-ade6-ba7a26a99a38" containerName="registry"
Oct 30 12:33:26.712734 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.712735    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67405c0f-c406-4e55-ade6-ba7a26a99a38" containerName="registry"
Oct 30 12:33:26.712859 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.712757    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fcbd57e7-45d9-472f-9afe-ded1285276b9" containerName="registry"
Oct 30 12:33:26.712859 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.712767    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbd57e7-45d9-472f-9afe-ded1285276b9" containerName="registry"
Oct 30 12:33:26.712859 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.712843    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbd57e7-45d9-472f-9afe-ded1285276b9" containerName="registry"
Oct 30 12:33:26.713050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.712860    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="67405c0f-c406-4e55-ade6-ba7a26a99a38" containerName="registry"
Oct 30 12:33:26.716031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.716008    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.736556 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.736191    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tnq7c-6db6s"]
Oct 30 12:33:26.738467 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.738435    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-trusted-ca-bundle\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.738582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.738480    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-archives-path\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.738582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.738518    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-service-ca-bundle\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.738582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.738550    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mh4\" (UniqueName: \"kubernetes.io/projected/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-kube-api-access-d6mh4\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.815779 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.815752    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:33:26.839778 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.839742    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path\") pod \"8c2ec818-97ad-4634-9c26-d1f6803d439d\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") "
Oct 30 12:33:26.839934 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.839805    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle\") pod \"8c2ec818-97ad-4634-9c26-d1f6803d439d\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") "
Oct 30 12:33:26.839934 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.839830    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb\" (UniqueName: \"kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb\") pod \"8c2ec818-97ad-4634-9c26-d1f6803d439d\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") "
Oct 30 12:33:26.839934 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.839876    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle\") pod \"8c2ec818-97ad-4634-9c26-d1f6803d439d\" (UID: \"8c2ec818-97ad-4634-9c26-d1f6803d439d\") "
Oct 30 12:33:26.840092 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840031    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-trusted-ca-bundle\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.840092 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840077    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-archives-path\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.840197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840068    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path" pod "8c2ec818-97ad-4634-9c26-d1f6803d439d" (UID: "8c2ec818-97ad-4634-9c26-d1f6803d439d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:33:26.840197 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840127    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-service-ca-bundle\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.840326 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840181    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mh4\" (UniqueName: \"kubernetes.io/projected/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-kube-api-access-d6mh4\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.840326 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840301    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle" pod "8c2ec818-97ad-4634-9c26-d1f6803d439d" (UID: "8c2ec818-97ad-4634-9c26-d1f6803d439d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:26.840432 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840357    4993 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8c2ec818-97ad-4634-9c26-d1f6803d439d-archives-path\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:26.840432 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840389    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle" pod "8c2ec818-97ad-4634-9c26-d1f6803d439d" (UID: "8c2ec818-97ad-4634-9c26-d1f6803d439d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:26.840599 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840571    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-archives-path\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.840913 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.840896    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-service-ca-bundle\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.841293 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.841275    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-trusted-ca-bundle\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.842414 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.842390    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb" pod "8c2ec818-97ad-4634-9c26-d1f6803d439d" (UID: "8c2ec818-97ad-4634-9c26-d1f6803d439d"). InnerVolumeSpecName "kube-api-access-xjrfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:26.865302 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.865268    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mh4\" (UniqueName: \"kubernetes.io/projected/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-kube-api-access-d6mh4\") pod \"periodic-gathering-tnq7c-6db6s\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") " pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:26.940972 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.940928    4993 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-service-ca-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:26.940972 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.940963    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrfb\" (UniqueName: \"kubernetes.io/projected/8c2ec818-97ad-4634-9c26-d1f6803d439d-kube-api-access-xjrfb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:26.940972 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:26.940973    4993 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2ec818-97ad-4634-9c26-d1f6803d439d-trusted-ca-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:33:27.029147 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.029106    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:33:27.105353 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.105323    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7tvl5" event={"ID":"929ccbc1-e13f-4dba-b00f-11a9c96d62f8","Type":"ContainerStarted","Data":"c8f36bc7dcd1f8efca790516c749a61236ea87101531a92079a7d252d57f2b4e"}
Oct 30 12:33:27.113436 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.113397    4993 generic.go:356] "Generic (PLEG): container finished" podID="8c2ec818-97ad-4634-9c26-d1f6803d439d" containerID="fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da" exitCode=2
Oct 30 12:33:27.113584 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.113543    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zks2p-9688t" event={"ID":"8c2ec818-97ad-4634-9c26-d1f6803d439d","Type":"ContainerDied","Data":"fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da"}
Oct 30 12:33:27.113633 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.113619    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zks2p-9688t" event={"ID":"8c2ec818-97ad-4634-9c26-d1f6803d439d","Type":"ContainerDied","Data":"720d4f499c17366e77d1fd53d62ad623a985e0656585df587066320bf011490a"}
Oct 30 12:33:27.113668 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.113648    4993 scope.go:122] "RemoveContainer" containerID="fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da"
Oct 30 12:33:27.113808 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.113793    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zks2p-9688t"
Oct 30 12:33:27.116291 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.116264    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbk6k" event={"ID":"a04059dd-fc63-4523-829f-c0ce3c4b8129","Type":"ContainerStarted","Data":"668c07b5d645abacc2c66142c7d32ec2f4f99a28896567d97ada5d8557051792"}
Oct 30 12:33:27.123122 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.123101    4993 scope.go:122] "RemoveContainer" containerID="fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da"
Oct 30 12:33:27.123461 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:33:27.123440    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da\": container with ID starting with fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da not found: ID does not exist" containerID="fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da"
Oct 30 12:33:27.123536 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.123473    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da"} err="failed to get container status \"fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da\": rpc error: code = NotFound desc = could not find container \"fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da\": container with ID starting with fb3123456f486b6b7786a43a6c4a726cb19da71b93b5410419cf8f14b27993da not found: ID does not exist"
Oct 30 12:33:27.224873 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.224839    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-zks2p-9688t"]
Oct 30 12:33:27.228594 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.228564    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-zks2p-9688t"]
Oct 30 12:33:27.321705 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.321669    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tnq7c-6db6s"]
Oct 30 12:33:27.325262 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:33:27.325233    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ac2d42_3f8c_4bbf_a9f2_b7059a65d009.slice/crio-51d6bfa2cc9302d9ff9102d16a405fccb221da313c743d489885d780723cebad WatchSource:0}: Error finding container 51d6bfa2cc9302d9ff9102d16a405fccb221da313c743d489885d780723cebad: Status 404 returned error can't find the container with id 51d6bfa2cc9302d9ff9102d16a405fccb221da313c743d489885d780723cebad
Oct 30 12:33:27.689745 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.689650    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2ec818-97ad-4634-9c26-d1f6803d439d" path="/var/lib/kubelet/pods/8c2ec818-97ad-4634-9c26-d1f6803d439d/volumes"
Oct 30 12:33:27.826850 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:27.826812    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77bb4759d7-ztvnj_51f4f667-c671-472c-ae68-beb10d17b645/router/0.log"
Oct 30 12:33:28.004072 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:28.004039    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vd2n6_b60e34dd-a30c-4f35-9593-1935139f9a47/serve-healthcheck-canary/0.log"
Oct 30 12:33:28.123156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:28.123110    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tnq7c-6db6s" event={"ID":"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009","Type":"ContainerStarted","Data":"4134e2d9ddaf030f6171d8fc84b0297e4c9802149d8f7ff4ea533ca8ccc6020c"}
Oct 30 12:33:28.123599 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:28.123171    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tnq7c-6db6s" event={"ID":"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009","Type":"ContainerStarted","Data":"51d6bfa2cc9302d9ff9102d16a405fccb221da313c743d489885d780723cebad"}
Oct 30 12:33:28.162716 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:28.162659    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-tnq7c-6db6s" podStartSLOduration=2.162644197 podStartE2EDuration="2.162644197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:28.161949415 +0000 UTC m=+72.637669023" watchObservedRunningTime="2026-10-30 12:33:28.162644197 +0000 UTC m=+72.638363801"
Oct 30 12:33:38.567095 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:33:38.567052    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:33:46.193997 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:46.193959    4993 generic.go:356] "Generic (PLEG): container finished" podID="bddd1e71-4116-437a-b7ee-873bc218fa0b" containerID="5505c86ca71a58b5f3db3081fcfa0f13be7102588f17a48478e1a11e111e2a4a" exitCode=0
Oct 30 12:33:46.194573 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:46.194036    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs" event={"ID":"bddd1e71-4116-437a-b7ee-873bc218fa0b","Type":"ContainerDied","Data":"5505c86ca71a58b5f3db3081fcfa0f13be7102588f17a48478e1a11e111e2a4a"}
Oct 30 12:33:46.194573 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:46.194493    4993 scope.go:122] "RemoveContainer" containerID="5505c86ca71a58b5f3db3081fcfa0f13be7102588f17a48478e1a11e111e2a4a"
Oct 30 12:33:47.200372 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:33:47.200339    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r2wrs" event={"ID":"bddd1e71-4116-437a-b7ee-873bc218fa0b","Type":"ContainerStarted","Data":"9a481f06f5cf2d3bc1b7c9f2a6161fa1b3a8979b320a769b3ad2548023527aa4"}
Oct 30 12:34:08.568332 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:34:08.568288    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:34:38.568646 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:34:38.568600    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:34:52.464995 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:52.464907    4993 generic.go:356] "Generic (PLEG): container finished" podID="a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009" containerID="4134e2d9ddaf030f6171d8fc84b0297e4c9802149d8f7ff4ea533ca8ccc6020c" exitCode=0
Oct 30 12:34:52.464995 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:52.464948    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tnq7c-6db6s" event={"ID":"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009","Type":"ContainerDied","Data":"4134e2d9ddaf030f6171d8fc84b0297e4c9802149d8f7ff4ea533ca8ccc6020c"}
Oct 30 12:34:53.616542 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.616497    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:34:53.734863 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.734826    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-kube-api-access-d6mh4\" (UniqueName: \"kubernetes.io/projected/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-kube-api-access-d6mh4\") pod \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") "
Oct 30 12:34:53.735199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.734883    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-archives-path\") pod \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") "
Oct 30 12:34:53.735199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.734933    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-service-ca-bundle\") pod \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") "
Oct 30 12:34:53.735199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.734973    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-trusted-ca-bundle\") pod \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\" (UID: \"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009\") "
Oct 30 12:34:53.735434 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.735306    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-service-ca-bundle" pod "a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009" (UID: "a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:53.735434 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.735279    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-archives-path" pod "a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009" (UID: "a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:53.735537 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.735481    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-trusted-ca-bundle" pod "a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009" (UID: "a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:53.737068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.737049    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-kube-api-access-d6mh4" pod "a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009" (UID: "a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009"). InnerVolumeSpecName "kube-api-access-d6mh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:53.835732 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.835696    4993 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-service-ca-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:34:53.835732 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.835725    4993 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-trusted-ca-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:34:53.835732 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.835735    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6mh4\" (UniqueName: \"kubernetes.io/projected/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-kube-api-access-d6mh4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:34:53.835975 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:53.835746    4993 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009-archives-path\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:34:54.474100 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:54.474066    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tnq7c-6db6s"
Oct 30 12:34:54.474300 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:54.474065    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tnq7c-6db6s" event={"ID":"a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009","Type":"ContainerDied","Data":"51d6bfa2cc9302d9ff9102d16a405fccb221da313c743d489885d780723cebad"}
Oct 30 12:34:54.474300 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:34:54.474195    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51d6bfa2cc9302d9ff9102d16a405fccb221da313c743d489885d780723cebad"
Oct 30 12:35:08.567851 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:35:08.567806    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:35:38.567937 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:35:38.567890    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:36:08.566760 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:36:08.566724    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:36:38.568360 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:36:38.568316    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:37:08.567992 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:37:08.567949    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:37:14.690312 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.690273    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"]
Oct 30 12:37:14.690825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.690607    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009" containerName="insights-gathering"
Oct 30 12:37:14.690825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.690620    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009" containerName="insights-gathering"
Oct 30 12:37:14.690825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.690641    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8c2ec818-97ad-4634-9c26-d1f6803d439d" containerName="insights-gathering"
Oct 30 12:37:14.690825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.690646    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2ec818-97ad-4634-9c26-d1f6803d439d" containerName="insights-gathering"
Oct 30 12:37:14.690825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.690702    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ac2d42-3f8c-4bbf-a9f2-b7059a65d009" containerName="insights-gathering"
Oct 30 12:37:14.690825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.690713    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2ec818-97ad-4634-9c26-d1f6803d439d" containerName="insights-gathering"
Oct 30 12:37:14.693904 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.693884    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.709137 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.709102    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 12:37:14.721246 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.721195    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 12:37:14.721435 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.721232    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-xlrqn\""
Oct 30 12:37:14.722553 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.722527    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"]
Oct 30 12:37:14.731149 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.731116    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.731338 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.731158    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdf9\" (UniqueName: \"kubernetes.io/projected/2827150c-2b55-41b4-a73e-b20450665e15-kube-api-access-5hdf9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.731338 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.731183    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.831869 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.831823    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.831869 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.831872    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdf9\" (UniqueName: \"kubernetes.io/projected/2827150c-2b55-41b4-a73e-b20450665e15-kube-api-access-5hdf9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.832185 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.831908    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.832331 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.832308    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.832391 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.832339    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:14.856646 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:14.856605    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdf9\" (UniqueName: \"kubernetes.io/projected/2827150c-2b55-41b4-a73e-b20450665e15-kube-api-access-5hdf9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:15.014712 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:15.014681    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:15.308881 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:15.308780    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"]
Oct 30 12:37:15.311810 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:15.311775    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2827150c_2b55_41b4_a73e_b20450665e15.slice/crio-3d0b05d8f589a68441f604c48855d330ca0cf26b60009d29b4faf4663faa150c WatchSource:0}: Error finding container 3d0b05d8f589a68441f604c48855d330ca0cf26b60009d29b4faf4663faa150c: Status 404 returned error can't find the container with id 3d0b05d8f589a68441f604c48855d330ca0cf26b60009d29b4faf4663faa150c
Oct 30 12:37:15.707750 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:15.707660    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:37:15.708375 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:15.708354    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:37:16.019726 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.019683    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2" event={"ID":"2827150c-2b55-41b4-a73e-b20450665e15","Type":"ContainerStarted","Data":"3d0b05d8f589a68441f604c48855d330ca0cf26b60009d29b4faf4663faa150c"}
Oct 30 12:37:16.143105 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.143052    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"]
Oct 30 12:37:16.148311 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.148276    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.177737 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.177703    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kfp5d\""
Oct 30 12:37:16.179047 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.179010    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"]
Oct 30 12:37:16.243938 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.243899    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"]
Oct 30 12:37:16.248098 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.248071    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.248798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.248758    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrvjv\" (UniqueName: \"kubernetes.io/projected/71de8d5d-ef64-47cc-a616-604fba27de01-kube-api-access-jrvjv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.248940 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.248825    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.249014 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.248978    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.266860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.266831    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-czx8d\""
Oct 30 12:37:16.268931 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.268895    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"]
Oct 30 12:37:16.338337 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.338247    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"]
Oct 30 12:37:16.342187 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.342156    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.349765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.349719    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.349949 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.349778    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.349949 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.349836    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.349949 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.349883    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.350488 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.349926    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrvjv\" (UniqueName: \"kubernetes.io/projected/71de8d5d-ef64-47cc-a616-604fba27de01-kube-api-access-jrvjv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.350488 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.350031    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwrh\" (UniqueName: \"kubernetes.io/projected/2f2c433f-eed8-48c1-b2aa-e46df5b91887-kube-api-access-sjwrh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.350488 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.350181    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.350488 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.350437    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.357314 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.357273    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-896q8\""
Oct 30 12:37:16.362501 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.362436    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"]
Oct 30 12:37:16.374317 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.374281    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrvjv\" (UniqueName: \"kubernetes.io/projected/71de8d5d-ef64-47cc-a616-604fba27de01-kube-api-access-jrvjv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.442635 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.442593    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"]
Oct 30 12:37:16.446559 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.446533    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.450504 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.450475    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.450665 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.450519    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.450665 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.450551    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.450665 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.450593    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.450665 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.450642    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwrh\" (UniqueName: \"kubernetes.io/projected/2f2c433f-eed8-48c1-b2aa-e46df5b91887-kube-api-access-sjwrh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.450892 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.450722    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hws\" (UniqueName: \"kubernetes.io/projected/19e79f7e-972d-4b12-8b33-ff4108d9bd37-kube-api-access-88hws\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.451181 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.451155    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.451327 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.451283    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.461423 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.461397    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lwv6w\""
Oct 30 12:37:16.467287 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.467247    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"]
Oct 30 12:37:16.470595 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.470553    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:16.476151 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.476123    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwrh\" (UniqueName: \"kubernetes.io/projected/2f2c433f-eed8-48c1-b2aa-e46df5b91887-kube-api-access-sjwrh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.551765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.551715    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.551936 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.551790    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnxv\" (UniqueName: \"kubernetes.io/projected/9fc3beff-53a6-42d4-b8d9-54acf279ab51-kube-api-access-6vnxv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.551936 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.551825    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hws\" (UniqueName: \"kubernetes.io/projected/19e79f7e-972d-4b12-8b33-ff4108d9bd37-kube-api-access-88hws\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.551936 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.551858    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.552329 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.552297    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.552431 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.552352    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.552480 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.552310    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.552542 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.552532    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.566814 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.566672    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:16.583856 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.583818    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hws\" (UniqueName: \"kubernetes.io/projected/19e79f7e-972d-4b12-8b33-ff4108d9bd37-kube-api-access-88hws\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.653593 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.653561    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnxv\" (UniqueName: \"kubernetes.io/projected/9fc3beff-53a6-42d4-b8d9-54acf279ab51-kube-api-access-6vnxv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.653785 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.653614    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.653785 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.653691    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.654104 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.654082    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.654188 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.654111    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.660600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.660563    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:16.677972 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.677934    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnxv\" (UniqueName: \"kubernetes.io/projected/9fc3beff-53a6-42d4-b8d9-54acf279ab51-kube-api-access-6vnxv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.765623 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.765582    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:16.781699 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.781663    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"]
Oct 30 12:37:16.784269 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:16.784183    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71de8d5d_ef64_47cc_a616_604fba27de01.slice/crio-2cf455f10b2285929a617939682bed1aaf1f10e4563f695e4d2677754d705e18 WatchSource:0}: Error finding container 2cf455f10b2285929a617939682bed1aaf1f10e4563f695e4d2677754d705e18: Status 404 returned error can't find the container with id 2cf455f10b2285929a617939682bed1aaf1f10e4563f695e4d2677754d705e18
Oct 30 12:37:16.876842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.876808    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"]
Oct 30 12:37:16.880907 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:16.880867    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2c433f_eed8_48c1_b2aa_e46df5b91887.slice/crio-5931c3dbccc85c4957536343c396b47868b990c83dcdee82127eb6b8daf839b2 WatchSource:0}: Error finding container 5931c3dbccc85c4957536343c396b47868b990c83dcdee82127eb6b8daf839b2: Status 404 returned error can't find the container with id 5931c3dbccc85c4957536343c396b47868b990c83dcdee82127eb6b8daf839b2
Oct 30 12:37:16.980010 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:16.979913    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"]
Oct 30 12:37:16.983368 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:16.983335    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e79f7e_972d_4b12_8b33_ff4108d9bd37.slice/crio-263c8a8e757da3492151455fb5ec188442fc4484a412dee57d5cdcb671b55179 WatchSource:0}: Error finding container 263c8a8e757da3492151455fb5ec188442fc4484a412dee57d5cdcb671b55179: Status 404 returned error can't find the container with id 263c8a8e757da3492151455fb5ec188442fc4484a412dee57d5cdcb671b55179
Oct 30 12:37:17.025960 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.025921    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x" event={"ID":"19e79f7e-972d-4b12-8b33-ff4108d9bd37","Type":"ContainerStarted","Data":"263c8a8e757da3492151455fb5ec188442fc4484a412dee57d5cdcb671b55179"}
Oct 30 12:37:17.027385 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.027279    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk" event={"ID":"2f2c433f-eed8-48c1-b2aa-e46df5b91887","Type":"ContainerStarted","Data":"5931c3dbccc85c4957536343c396b47868b990c83dcdee82127eb6b8daf839b2"}
Oct 30 12:37:17.028708 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.028681    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc" event={"ID":"71de8d5d-ef64-47cc-a616-604fba27de01","Type":"ContainerStarted","Data":"2cf455f10b2285929a617939682bed1aaf1f10e4563f695e4d2677754d705e18"}
Oct 30 12:37:17.077543 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.077332    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"]
Oct 30 12:37:17.080361 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:17.080328    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc3beff_53a6_42d4_b8d9_54acf279ab51.slice/crio-07e7084aa091154d17681b7cb020586fd6bf46b1c7f64425b4fa7ff924c443b4 WatchSource:0}: Error finding container 07e7084aa091154d17681b7cb020586fd6bf46b1c7f64425b4fa7ff924c443b4: Status 404 returned error can't find the container with id 07e7084aa091154d17681b7cb020586fd6bf46b1c7f64425b4fa7ff924c443b4
Oct 30 12:37:17.912154 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.912106    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"]
Oct 30 12:37:17.916491 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.916462    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:17.932165 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.932127    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-lgfsd\""
Oct 30 12:37:17.940463 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.940424    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"]
Oct 30 12:37:17.965559 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.965483    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:17.965753 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.965686    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:17.965827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:17.965756    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nh6n\" (UniqueName: \"kubernetes.io/projected/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-kube-api-access-5nh6n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:18.036699 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.036659    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b" event={"ID":"9fc3beff-53a6-42d4-b8d9-54acf279ab51","Type":"ContainerStarted","Data":"07e7084aa091154d17681b7cb020586fd6bf46b1c7f64425b4fa7ff924c443b4"}
Oct 30 12:37:18.067237 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.066969    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:18.067237 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.067024    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nh6n\" (UniqueName: \"kubernetes.io/projected/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-kube-api-access-5nh6n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:18.067237 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.067118    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:18.067538 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.067479    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:18.067915 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.067867    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:18.091404 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.091335    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nh6n\" (UniqueName: \"kubernetes.io/projected/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-kube-api-access-5nh6n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:18.140981 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.139226    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"]
Oct 30 12:37:18.144734 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.143874    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.160756 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.160706    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tcq9j\""
Oct 30 12:37:18.166671 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.166577    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168530    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168564    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168593    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168614    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168640    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168656    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33694bd-a804-4182-8d4b-e30156066d8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168678    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33694bd-a804-4182-8d4b-e30156066d8f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168698    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pqrm\" (UniqueName: \"kubernetes.io/projected/d33694bd-a804-4182-8d4b-e30156066d8f-kube-api-access-7pqrm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.168849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.168722    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33694bd-a804-4182-8d4b-e30156066d8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.170597 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.170445    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"]
Oct 30 12:37:18.242319 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.242022    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269627    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269678    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33694bd-a804-4182-8d4b-e30156066d8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269722    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33694bd-a804-4182-8d4b-e30156066d8f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269757    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqrm\" (UniqueName: \"kubernetes.io/projected/d33694bd-a804-4182-8d4b-e30156066d8f-kube-api-access-7pqrm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269808    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33694bd-a804-4182-8d4b-e30156066d8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269851    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269887    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269941    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.269977    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.270454    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.272152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.271452    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d33694bd-a804-4182-8d4b-e30156066d8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.274398 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.273172    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.274398 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.273427    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.274398 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.273641    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.275760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.275700    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d33694bd-a804-4182-8d4b-e30156066d8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.282097 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.282059    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"]
Oct 30 12:37:18.284812 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.284785    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d33694bd-a804-4182-8d4b-e30156066d8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.291524 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.291493    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.300314 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.299697    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"]
Oct 30 12:37:18.303701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.303570    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqrm\" (UniqueName: \"kubernetes.io/projected/d33694bd-a804-4182-8d4b-e30156066d8f-kube-api-access-7pqrm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.307246 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.305895    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.307246 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.305928    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d33694bd-a804-4182-8d4b-e30156066d8f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgkz8\" (UID: \"d33694bd-a804-4182-8d4b-e30156066d8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.318583 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.318547    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-k7zw7\""
Oct 30 12:37:18.320797 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.320715    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"]
Oct 30 12:37:18.339954 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.339919    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"]
Oct 30 12:37:18.370820 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.370779    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.371030 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.370838    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.371030 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.370893    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.371030 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.370925    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.371030 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.371007    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fg84\" (UniqueName: \"kubernetes.io/projected/3796532e-93ff-4a27-9c23-187bcb25b9d5-kube-api-access-2fg84\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.371284 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.371056    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.371284 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.371119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.371284 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.371189    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.371447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.371294    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.371447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.371331    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bb41a5-08e3-4e05-baee-c015d7239d96-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.371447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.371369    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.371447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.371399    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgwf\" (UniqueName: \"kubernetes.io/projected/81bb41a5-08e3-4e05-baee-c015d7239d96-kube-api-access-gpgwf\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.415538 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.415501    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-fnvgx\""
Oct 30 12:37:18.464091 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.463996    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:18.472302 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.472251    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.473076 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473050    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.473245 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.472323    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.473245 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473197    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.473373 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473244    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fg84\" (UniqueName: \"kubernetes.io/projected/3796532e-93ff-4a27-9c23-187bcb25b9d5-kube-api-access-2fg84\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.473373 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473273    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.473373 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473303    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.473373 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473330    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.473373 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473369    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.473619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473396    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bb41a5-08e3-4e05-baee-c015d7239d96-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.473619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473438    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.473619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473472    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgwf\" (UniqueName: \"kubernetes.io/projected/81bb41a5-08e3-4e05-baee-c015d7239d96-kube-api-access-gpgwf\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.473619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.473559    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.474424 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.474401    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.474696 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.474654    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.474941 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.474922    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.476250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.476227    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bb41a5-08e3-4e05-baee-c015d7239d96-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.476535 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.476509    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.476788 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.476767    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.479328 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.479298    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.480252 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.480194    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.509917 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.508844    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fg84\" (UniqueName: \"kubernetes.io/projected/3796532e-93ff-4a27-9c23-187bcb25b9d5-kube-api-access-2fg84\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.510617 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.510572    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgwf\" (UniqueName: \"kubernetes.io/projected/81bb41a5-08e3-4e05-baee-c015d7239d96-kube-api-access-gpgwf\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.511044 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.511010    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bb41a5-08e3-4e05-baee-c015d7239d96-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh\" (UID: \"81bb41a5-08e3-4e05-baee-c015d7239d96\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.582770 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.582736    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"]
Oct 30 12:37:18.583363 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:18.583333    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1ddcdc_cf61_4c62_bce2_36b0bc4f3376.slice/crio-c45eb739ff8612c5284820b5fb8161a5921b3b01abe4badfefcd1cfb4d356365 WatchSource:0}: Error finding container c45eb739ff8612c5284820b5fb8161a5921b3b01abe4badfefcd1cfb4d356365: Status 404 returned error can't find the container with id c45eb739ff8612c5284820b5fb8161a5921b3b01abe4badfefcd1cfb4d356365
Oct 30 12:37:18.591826 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.591799    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:37:18.643381 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.643338    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:18.649329 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.649300    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:18.811850 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.811658    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"]
Oct 30 12:37:18.821235 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:18.821188    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33694bd_a804_4182_8d4b_e30156066d8f.slice/crio-fa07e2439f383a84f62c5515ade46555174fd146e9778853719599214be4e0cc WatchSource:0}: Error finding container fa07e2439f383a84f62c5515ade46555174fd146e9778853719599214be4e0cc: Status 404 returned error can't find the container with id fa07e2439f383a84f62c5515ade46555174fd146e9778853719599214be4e0cc
Oct 30 12:37:18.977752 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:18.977523    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"]
Oct 30 12:37:18.981913 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:18.981833    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3796532e_93ff_4a27_9c23_187bcb25b9d5.slice/crio-0992ec800fbde34ae8f32e13d74088b4e3178a7fbca1c0c61bb939ea7ebfe170 WatchSource:0}: Error finding container 0992ec800fbde34ae8f32e13d74088b4e3178a7fbca1c0c61bb939ea7ebfe170: Status 404 returned error can't find the container with id 0992ec800fbde34ae8f32e13d74088b4e3178a7fbca1c0c61bb939ea7ebfe170
Oct 30 12:37:19.002576 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:19.002271    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"]
Oct 30 12:37:19.005916 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:19.005871    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bb41a5_08e3_4e05_baee_c015d7239d96.slice/crio-1a25fab77cd4bd230f594373c4029cb007257e31cd1631326adf9cc564ccac1e WatchSource:0}: Error finding container 1a25fab77cd4bd230f594373c4029cb007257e31cd1631326adf9cc564ccac1e: Status 404 returned error can't find the container with id 1a25fab77cd4bd230f594373c4029cb007257e31cd1631326adf9cc564ccac1e
Oct 30 12:37:19.051438 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:19.051391    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx" event={"ID":"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376","Type":"ContainerStarted","Data":"c45eb739ff8612c5284820b5fb8161a5921b3b01abe4badfefcd1cfb4d356365"}
Oct 30 12:37:19.056097 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:19.056013    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh" event={"ID":"81bb41a5-08e3-4e05-baee-c015d7239d96","Type":"ContainerStarted","Data":"1a25fab77cd4bd230f594373c4029cb007257e31cd1631326adf9cc564ccac1e"}
Oct 30 12:37:19.061934 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:19.061844    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd" event={"ID":"3796532e-93ff-4a27-9c23-187bcb25b9d5","Type":"ContainerStarted","Data":"0992ec800fbde34ae8f32e13d74088b4e3178a7fbca1c0c61bb939ea7ebfe170"}
Oct 30 12:37:19.077092 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:19.076948    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" event={"ID":"d33694bd-a804-4182-8d4b-e30156066d8f","Type":"ContainerStarted","Data":"fa07e2439f383a84f62c5515ade46555174fd146e9778853719599214be4e0cc"}
Oct 30 12:37:26.944601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:26.944083    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:37:26.944601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:26.944171    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:37:26.944601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:26.944241    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:37:27.149769 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:27.148761    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" event={"ID":"d33694bd-a804-4182-8d4b-e30156066d8f","Type":"ContainerStarted","Data":"d21bc8d958ecbc5231dc2152a9a388174a8a85f223f853be8f053aaf2878fc0c"}
Oct 30 12:37:27.220428 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:27.219467    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podStartSLOduration=1.101819998 podStartE2EDuration="9.21944811s" totalImagesPullingTime="8.117628112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:27.206711063 +0000 UTC m=+311.682430674" watchObservedRunningTime="2026-10-30 12:37:27.21944811 +0000 UTC m=+311.695167716"
Oct 30 12:37:27.419258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:27.419169    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:37:27.419406 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:27.419278    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:37:27.419406 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:27.419310    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:37:27.464937 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:27.464903    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:28.167424 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:28.167354    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh" event={"ID":"81bb41a5-08e3-4e05-baee-c015d7239d96","Type":"ContainerStarted","Data":"630fc5d792345669d11d15270c5d4f366efa47cc4e22273cb5b3af17becf09f4"}
Oct 30 12:37:28.238701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:28.237783    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh" podStartSLOduration=1.832182253 podStartE2EDuration="10.23776752s" totalImagesPullingTime="8.405585267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:28.224578181 +0000 UTC m=+312.700297815" watchObservedRunningTime="2026-10-30 12:37:28.23776752 +0000 UTC m=+312.713487131"
Oct 30 12:37:28.471109 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:28.471016    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:28.649929 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:28.649887    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:28.655944 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:28.655906    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:29.195049 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:29.194155    4993 generic.go:356] "Generic (PLEG): container finished" podID="2827150c-2b55-41b4-a73e-b20450665e15" containerID="96e8cfe52a67e975eb3f4eb250eda18fe60119f2b8b2458ff847ccbf4ec3fe63" exitCode=0
Oct 30 12:37:29.195049 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:29.194293    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2" event={"ID":"2827150c-2b55-41b4-a73e-b20450665e15","Type":"ContainerDied","Data":"96e8cfe52a67e975eb3f4eb250eda18fe60119f2b8b2458ff847ccbf4ec3fe63"}
Oct 30 12:37:29.195049 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:29.194767    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:29.195049 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:29.194814    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:29.195975 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:29.195878    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh"
Oct 30 12:37:29.196790 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:29.196711    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:37:30.208992 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:30.208927    4993 generic.go:356] "Generic (PLEG): container finished" podID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerID="0b871124e57962e2c3b34b10e2bfc40f874cae8e40cc85c4ea381af20dba4933" exitCode=0
Oct 30 12:37:30.210884 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:30.210852    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd" event={"ID":"3796532e-93ff-4a27-9c23-187bcb25b9d5","Type":"ContainerDied","Data":"0b871124e57962e2c3b34b10e2bfc40f874cae8e40cc85c4ea381af20dba4933"}
Oct 30 12:37:32.229275 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:32.228278    4993 generic.go:356] "Generic (PLEG): container finished" podID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerID="0c176ff867ef05b091de71377aee37c9d6752e41aa9e13c20bfabbf33339f0de" exitCode=0
Oct 30 12:37:32.229275 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:32.228339    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd" event={"ID":"3796532e-93ff-4a27-9c23-187bcb25b9d5","Type":"ContainerDied","Data":"0c176ff867ef05b091de71377aee37c9d6752e41aa9e13c20bfabbf33339f0de"}
Oct 30 12:37:33.236217 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.236156    4993 generic.go:356] "Generic (PLEG): container finished" podID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerID="807d147cf80233f9bfa317f82d56c94e5dce2dcfc75be7d9e5dc8a102afe7629" exitCode=0
Oct 30 12:37:33.236645 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.236234    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x" event={"ID":"19e79f7e-972d-4b12-8b33-ff4108d9bd37","Type":"ContainerDied","Data":"807d147cf80233f9bfa317f82d56c94e5dce2dcfc75be7d9e5dc8a102afe7629"}
Oct 30 12:37:33.238443 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.238409    4993 generic.go:356] "Generic (PLEG): container finished" podID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerID="5c3f8eb83187232183320f48b678736fe87c0b51b2c52fdead2ccf6dfbfe0040" exitCode=0
Oct 30 12:37:33.238584 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.238511    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk" event={"ID":"2f2c433f-eed8-48c1-b2aa-e46df5b91887","Type":"ContainerDied","Data":"5c3f8eb83187232183320f48b678736fe87c0b51b2c52fdead2ccf6dfbfe0040"}
Oct 30 12:37:33.240974 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.240920    4993 generic.go:356] "Generic (PLEG): container finished" podID="71de8d5d-ef64-47cc-a616-604fba27de01" containerID="dfd9aac4cdaa504e5af02b6b4931efb9f8152d659434ecf2e843ff1a98a86f6c" exitCode=0
Oct 30 12:37:33.241042 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.240959    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc" event={"ID":"71de8d5d-ef64-47cc-a616-604fba27de01","Type":"ContainerDied","Data":"dfd9aac4cdaa504e5af02b6b4931efb9f8152d659434ecf2e843ff1a98a86f6c"}
Oct 30 12:37:33.250022 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.249986    4993 generic.go:356] "Generic (PLEG): container finished" podID="2827150c-2b55-41b4-a73e-b20450665e15" containerID="b19344004a5f5e59e621be41f6b7e3c9f1d868493106e57c5599cab68258d839" exitCode=0
Oct 30 12:37:33.250136 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.250114    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2" event={"ID":"2827150c-2b55-41b4-a73e-b20450665e15","Type":"ContainerDied","Data":"b19344004a5f5e59e621be41f6b7e3c9f1d868493106e57c5599cab68258d839"}
Oct 30 12:37:33.252950 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.252915    4993 generic.go:356] "Generic (PLEG): container finished" podID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerID="cc6b45d97daa27213203f5ab0db61c78ab70a26bbe6312611df13595d127679a" exitCode=0
Oct 30 12:37:33.253142 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:33.252974    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b" event={"ID":"9fc3beff-53a6-42d4-b8d9-54acf279ab51","Type":"ContainerDied","Data":"cc6b45d97daa27213203f5ab0db61c78ab70a26bbe6312611df13595d127679a"}
Oct 30 12:37:34.268480 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.268444    4993 generic.go:356] "Generic (PLEG): container finished" podID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerID="35e97f308352c7379ac95ea4a1a00f56522e78083d1c2711f9ca021e713de8d2" exitCode=0
Oct 30 12:37:34.268965 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.268877    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b" event={"ID":"9fc3beff-53a6-42d4-b8d9-54acf279ab51","Type":"ContainerDied","Data":"35e97f308352c7379ac95ea4a1a00f56522e78083d1c2711f9ca021e713de8d2"}
Oct 30 12:37:34.273786 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.273756    4993 generic.go:356] "Generic (PLEG): container finished" podID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerID="8e88a1510fa4da01be8d5230bce484b5aefdab6e3d4e5cc4523a3c9b52ca405c" exitCode=0
Oct 30 12:37:34.274043 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.273877    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx" event={"ID":"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376","Type":"ContainerDied","Data":"8e88a1510fa4da01be8d5230bce484b5aefdab6e3d4e5cc4523a3c9b52ca405c"}
Oct 30 12:37:34.277422 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.277390    4993 generic.go:356] "Generic (PLEG): container finished" podID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerID="bbfad8a6e4a6e26c146ba6c4e3b0acf9fc2c79c56921a00f2e16216fff98e4e5" exitCode=0
Oct 30 12:37:34.277529 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.277475    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x" event={"ID":"19e79f7e-972d-4b12-8b33-ff4108d9bd37","Type":"ContainerDied","Data":"bbfad8a6e4a6e26c146ba6c4e3b0acf9fc2c79c56921a00f2e16216fff98e4e5"}
Oct 30 12:37:34.280330 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.280175    4993 generic.go:356] "Generic (PLEG): container finished" podID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerID="8852d45f95b418f4ab76a7f8f9c1f119739fe16b5e0b75436ada88a95bc3c719" exitCode=0
Oct 30 12:37:34.280330 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.280288    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk" event={"ID":"2f2c433f-eed8-48c1-b2aa-e46df5b91887","Type":"ContainerDied","Data":"8852d45f95b418f4ab76a7f8f9c1f119739fe16b5e0b75436ada88a95bc3c719"}
Oct 30 12:37:34.283442 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:34.283379    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc" event={"ID":"71de8d5d-ef64-47cc-a616-604fba27de01","Type":"ContainerStarted","Data":"969b99fdf8a5229ec86ed09d887066bb6a787d66b13cb59d55cff2a239314c56"}
Oct 30 12:37:35.296250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:35.296188    4993 generic.go:356] "Generic (PLEG): container finished" podID="71de8d5d-ef64-47cc-a616-604fba27de01" containerID="969b99fdf8a5229ec86ed09d887066bb6a787d66b13cb59d55cff2a239314c56" exitCode=0
Oct 30 12:37:35.296756 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:35.296351    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc" event={"ID":"71de8d5d-ef64-47cc-a616-604fba27de01","Type":"ContainerDied","Data":"969b99fdf8a5229ec86ed09d887066bb6a787d66b13cb59d55cff2a239314c56"}
Oct 30 12:37:38.587857 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:37:38.587782    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:37:40.349235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:40.349163    4993 generic.go:356] "Generic (PLEG): container finished" podID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerID="0ae470576f32d6aec4c1293967c45aea31648af7757614f77bd5205bd53ef14f" exitCode=0
Oct 30 12:37:40.349925 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:40.349839    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx" event={"ID":"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376","Type":"ContainerDied","Data":"0ae470576f32d6aec4c1293967c45aea31648af7757614f77bd5205bd53ef14f"}
Oct 30 12:37:45.410733 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:45.410123    4993 generic.go:356] "Generic (PLEG): container finished" podID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerID="66daa5b63ba3124857d304f7c1b9c05e22418945621826b6973c6137ea19fa8b" exitCode=0
Oct 30 12:37:45.410733 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:45.410218    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd" event={"ID":"3796532e-93ff-4a27-9c23-187bcb25b9d5","Type":"ContainerDied","Data":"66daa5b63ba3124857d304f7c1b9c05e22418945621826b6973c6137ea19fa8b"}
Oct 30 12:37:46.673714 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.673672    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:46.698603 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.698508    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3796532e-93ff-4a27-9c23-187bcb25b9d5-kube-api-access-2fg84\" (UniqueName: \"kubernetes.io/projected/3796532e-93ff-4a27-9c23-187bcb25b9d5-kube-api-access-2fg84\") pod \"3796532e-93ff-4a27-9c23-187bcb25b9d5\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") "
Oct 30 12:37:46.698798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.698642    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-bundle\") pod \"3796532e-93ff-4a27-9c23-187bcb25b9d5\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") "
Oct 30 12:37:46.698798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.698703    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-util\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-util\") pod \"3796532e-93ff-4a27-9c23-187bcb25b9d5\" (UID: \"3796532e-93ff-4a27-9c23-187bcb25b9d5\") "
Oct 30 12:37:46.704381 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.704338    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-bundle" pod "3796532e-93ff-4a27-9c23-187bcb25b9d5" (UID: "3796532e-93ff-4a27-9c23-187bcb25b9d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:46.709140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.709073    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-util" pod "3796532e-93ff-4a27-9c23-187bcb25b9d5" (UID: "3796532e-93ff-4a27-9c23-187bcb25b9d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:46.711955 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.711916    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3796532e-93ff-4a27-9c23-187bcb25b9d5-kube-api-access-2fg84" pod "3796532e-93ff-4a27-9c23-187bcb25b9d5" (UID: "3796532e-93ff-4a27-9c23-187bcb25b9d5"). InnerVolumeSpecName "kube-api-access-2fg84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:46.799908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.799861    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:46.799908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.799902    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3796532e-93ff-4a27-9c23-187bcb25b9d5-util\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:46.800165 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:46.799921    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fg84\" (UniqueName: \"kubernetes.io/projected/3796532e-93ff-4a27-9c23-187bcb25b9d5-kube-api-access-2fg84\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:47.456689 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:47.456603    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd" event={"ID":"3796532e-93ff-4a27-9c23-187bcb25b9d5","Type":"ContainerDied","Data":"0992ec800fbde34ae8f32e13d74088b4e3178a7fbca1c0c61bb939ea7ebfe170"}
Oct 30 12:37:47.456689 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:47.456657    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xv5cd"
Oct 30 12:37:47.456689 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:47.456662    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0992ec800fbde34ae8f32e13d74088b4e3178a7fbca1c0c61bb939ea7ebfe170"
Oct 30 12:37:48.465460 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:48.465281    4993 generic.go:356] "Generic (PLEG): container finished" podID="2827150c-2b55-41b4-a73e-b20450665e15" containerID="5e8f17ee5b9c43ea4e0c9e060a8bd75b0e2151f3a1e71a25d7e86eea789fed0a" exitCode=0
Oct 30 12:37:48.465460 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:48.465388    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2" event={"ID":"2827150c-2b55-41b4-a73e-b20450665e15","Type":"ContainerDied","Data":"5e8f17ee5b9c43ea4e0c9e060a8bd75b0e2151f3a1e71a25d7e86eea789fed0a"}
Oct 30 12:37:49.472064 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.471970    4993 generic.go:356] "Generic (PLEG): container finished" podID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerID="a973a17ff7add1a0ae261ee27e28c5cf753a646e1eb1675e6855f2cb5acd4899" exitCode=0
Oct 30 12:37:49.472534 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.472060    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b" event={"ID":"9fc3beff-53a6-42d4-b8d9-54acf279ab51","Type":"ContainerDied","Data":"a973a17ff7add1a0ae261ee27e28c5cf753a646e1eb1675e6855f2cb5acd4899"}
Oct 30 12:37:49.474262 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.474235    4993 generic.go:356] "Generic (PLEG): container finished" podID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerID="dbf3b2aa58028d354e5d58d23e5b944badd6f193979971af17f684286a161173" exitCode=0
Oct 30 12:37:49.474387 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.474308    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx" event={"ID":"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376","Type":"ContainerDied","Data":"dbf3b2aa58028d354e5d58d23e5b944badd6f193979971af17f684286a161173"}
Oct 30 12:37:49.476317 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.476293    4993 generic.go:356] "Generic (PLEG): container finished" podID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerID="fe16567f05b454f0e1f9778fffb89fdea228abd51d959d03a58c8b769083589e" exitCode=0
Oct 30 12:37:49.476438 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.476373    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x" event={"ID":"19e79f7e-972d-4b12-8b33-ff4108d9bd37","Type":"ContainerDied","Data":"fe16567f05b454f0e1f9778fffb89fdea228abd51d959d03a58c8b769083589e"}
Oct 30 12:37:49.478315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.478291    4993 generic.go:356] "Generic (PLEG): container finished" podID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerID="77475204a698dfd81a87605146f9d8a30a6466f8b6744fcad48ba47bf0498ae8" exitCode=0
Oct 30 12:37:49.478435 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.478406    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk" event={"ID":"2f2c433f-eed8-48c1-b2aa-e46df5b91887","Type":"ContainerDied","Data":"77475204a698dfd81a87605146f9d8a30a6466f8b6744fcad48ba47bf0498ae8"}
Oct 30 12:37:49.480281 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.480264    4993 generic.go:356] "Generic (PLEG): container finished" podID="71de8d5d-ef64-47cc-a616-604fba27de01" containerID="db650244cf17c0a16f395ade8e840187d61d0dee7ecf3eada4b340f79ae800d5" exitCode=0
Oct 30 12:37:49.480371 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.480318    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc" event={"ID":"71de8d5d-ef64-47cc-a616-604fba27de01","Type":"ContainerDied","Data":"db650244cf17c0a16f395ade8e840187d61d0dee7ecf3eada4b340f79ae800d5"}
Oct 30 12:37:49.597915 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.597889    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:49.626751 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.626714    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-bundle\") pod \"2827150c-2b55-41b4-a73e-b20450665e15\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") "
Oct 30 12:37:49.626902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.626837    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2827150c-2b55-41b4-a73e-b20450665e15-kube-api-access-5hdf9\" (UniqueName: \"kubernetes.io/projected/2827150c-2b55-41b4-a73e-b20450665e15-kube-api-access-5hdf9\") pod \"2827150c-2b55-41b4-a73e-b20450665e15\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") "
Oct 30 12:37:49.626902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.626868    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-util\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-util\") pod \"2827150c-2b55-41b4-a73e-b20450665e15\" (UID: \"2827150c-2b55-41b4-a73e-b20450665e15\") "
Oct 30 12:37:49.627265 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.627114    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-bundle" pod "2827150c-2b55-41b4-a73e-b20450665e15" (UID: "2827150c-2b55-41b4-a73e-b20450665e15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:49.628973 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.628948    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2827150c-2b55-41b4-a73e-b20450665e15-kube-api-access-5hdf9" pod "2827150c-2b55-41b4-a73e-b20450665e15" (UID: "2827150c-2b55-41b4-a73e-b20450665e15"). InnerVolumeSpecName "kube-api-access-5hdf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:49.632108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.632083    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-util" pod "2827150c-2b55-41b4-a73e-b20450665e15" (UID: "2827150c-2b55-41b4-a73e-b20450665e15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:49.727733 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.727691    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hdf9\" (UniqueName: \"kubernetes.io/projected/2827150c-2b55-41b4-a73e-b20450665e15-kube-api-access-5hdf9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:49.727733 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.727726    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-util\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:49.727733 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:49.727738    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2827150c-2b55-41b4-a73e-b20450665e15-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.486660 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.486614    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2" event={"ID":"2827150c-2b55-41b4-a73e-b20450665e15","Type":"ContainerDied","Data":"3d0b05d8f589a68441f604c48855d330ca0cf26b60009d29b4faf4663faa150c"}
Oct 30 12:37:50.486660 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.486644    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lc2k2"
Oct 30 12:37:50.487228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.486685    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d0b05d8f589a68441f604c48855d330ca0cf26b60009d29b4faf4663faa150c"
Oct 30 12:37:50.664846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.664814    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:50.719492 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.719467    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:50.722742 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.722713    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:50.726774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.726750    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:50.730120 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.730102    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:50.737239 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.737183    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc3beff-53a6-42d4-b8d9-54acf279ab51-kube-api-access-6vnxv\" (UniqueName: \"kubernetes.io/projected/9fc3beff-53a6-42d4-b8d9-54acf279ab51-kube-api-access-6vnxv\") pod \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") "
Oct 30 12:37:50.737307 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.737272    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-util\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-util\") pod \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") "
Oct 30 12:37:50.737307 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.737303    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-bundle\") pod \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\" (UID: \"9fc3beff-53a6-42d4-b8d9-54acf279ab51\") "
Oct 30 12:37:50.737797 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.737768    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-bundle" pod "9fc3beff-53a6-42d4-b8d9-54acf279ab51" (UID: "9fc3beff-53a6-42d4-b8d9-54acf279ab51"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.739426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.739399    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc3beff-53a6-42d4-b8d9-54acf279ab51-kube-api-access-6vnxv" pod "9fc3beff-53a6-42d4-b8d9-54acf279ab51" (UID: "9fc3beff-53a6-42d4-b8d9-54acf279ab51"). InnerVolumeSpecName "kube-api-access-6vnxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:50.742999 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.742978    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-util" pod "9fc3beff-53a6-42d4-b8d9-54acf279ab51" (UID: "9fc3beff-53a6-42d4-b8d9-54acf279ab51"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.838493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838451    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e79f7e-972d-4b12-8b33-ff4108d9bd37-kube-api-access-88hws\" (UniqueName: \"kubernetes.io/projected/19e79f7e-972d-4b12-8b33-ff4108d9bd37-kube-api-access-88hws\") pod \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") "
Oct 30 12:37:50.838694 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838500    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-bundle\") pod \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") "
Oct 30 12:37:50.838694 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838530    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-bundle\") pod \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") "
Oct 30 12:37:50.838694 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838580    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-util\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-util\") pod \"71de8d5d-ef64-47cc-a616-604fba27de01\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") "
Oct 30 12:37:50.838694 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838607    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2c433f-eed8-48c1-b2aa-e46df5b91887-kube-api-access-sjwrh\" (UniqueName: \"kubernetes.io/projected/2f2c433f-eed8-48c1-b2aa-e46df5b91887-kube-api-access-sjwrh\") pod \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") "
Oct 30 12:37:50.838694 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838645    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71de8d5d-ef64-47cc-a616-604fba27de01-kube-api-access-jrvjv\" (UniqueName: \"kubernetes.io/projected/71de8d5d-ef64-47cc-a616-604fba27de01-kube-api-access-jrvjv\") pod \"71de8d5d-ef64-47cc-a616-604fba27de01\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") "
Oct 30 12:37:50.838694 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838686    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-bundle\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-bundle\") pod \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") "
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838724    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-util\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-util\") pod \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") "
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838754    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-kube-api-access-5nh6n\" (UniqueName: \"kubernetes.io/projected/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-kube-api-access-5nh6n\") pod \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\" (UID: \"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376\") "
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838780    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-util\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-util\") pod \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\" (UID: \"2f2c433f-eed8-48c1-b2aa-e46df5b91887\") "
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838828    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-util\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-util\") pod \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\" (UID: \"19e79f7e-972d-4b12-8b33-ff4108d9bd37\") "
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.838878    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-bundle\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-bundle\") pod \"71de8d5d-ef64-47cc-a616-604fba27de01\" (UID: \"71de8d5d-ef64-47cc-a616-604fba27de01\") "
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.839168    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vnxv\" (UniqueName: \"kubernetes.io/projected/9fc3beff-53a6-42d4-b8d9-54acf279ab51-kube-api-access-6vnxv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.839187    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-util\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.839201    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fc3beff-53a6-42d4-b8d9-54acf279ab51-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.839198    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-bundle" pod "2f2c433f-eed8-48c1-b2aa-e46df5b91887" (UID: "2f2c433f-eed8-48c1-b2aa-e46df5b91887"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.839723    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-bundle" pod "5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" (UID: "5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.839961    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-bundle" pod "71de8d5d-ef64-47cc-a616-604fba27de01" (UID: "71de8d5d-ef64-47cc-a616-604fba27de01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.841183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.840780    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-bundle" pod "19e79f7e-972d-4b12-8b33-ff4108d9bd37" (UID: "19e79f7e-972d-4b12-8b33-ff4108d9bd37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.841823 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.841792    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e79f7e-972d-4b12-8b33-ff4108d9bd37-kube-api-access-88hws" pod "19e79f7e-972d-4b12-8b33-ff4108d9bd37" (UID: "19e79f7e-972d-4b12-8b33-ff4108d9bd37"). InnerVolumeSpecName "kube-api-access-88hws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:50.842219 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.842167    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2c433f-eed8-48c1-b2aa-e46df5b91887-kube-api-access-sjwrh" pod "2f2c433f-eed8-48c1-b2aa-e46df5b91887" (UID: "2f2c433f-eed8-48c1-b2aa-e46df5b91887"). InnerVolumeSpecName "kube-api-access-sjwrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:50.842546 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.842518    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-kube-api-access-5nh6n" pod "5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" (UID: "5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376"). InnerVolumeSpecName "kube-api-access-5nh6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:50.843126 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.843108    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71de8d5d-ef64-47cc-a616-604fba27de01-kube-api-access-jrvjv" pod "71de8d5d-ef64-47cc-a616-604fba27de01" (UID: "71de8d5d-ef64-47cc-a616-604fba27de01"). InnerVolumeSpecName "kube-api-access-jrvjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:50.846222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.846189    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-util" pod "5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" (UID: "5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.848312 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.848291    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-util" pod "2f2c433f-eed8-48c1-b2aa-e46df5b91887" (UID: "2f2c433f-eed8-48c1-b2aa-e46df5b91887"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.848429 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.848404    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-util" pod "71de8d5d-ef64-47cc-a616-604fba27de01" (UID: "71de8d5d-ef64-47cc-a616-604fba27de01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.849128 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.849109    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-util" pod "19e79f7e-972d-4b12-8b33-ff4108d9bd37" (UID: "19e79f7e-972d-4b12-8b33-ff4108d9bd37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:50.940365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940324    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-util\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940358    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwrh\" (UniqueName: \"kubernetes.io/projected/2f2c433f-eed8-48c1-b2aa-e46df5b91887-kube-api-access-sjwrh\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940375    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrvjv\" (UniqueName: \"kubernetes.io/projected/71de8d5d-ef64-47cc-a616-604fba27de01-kube-api-access-jrvjv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940387    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940396    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-util\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940403    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nh6n\" (UniqueName: \"kubernetes.io/projected/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-kube-api-access-5nh6n\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940411    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-util\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940420    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19e79f7e-972d-4b12-8b33-ff4108d9bd37-util\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940428    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71de8d5d-ef64-47cc-a616-604fba27de01-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940436    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hws\" (UniqueName: \"kubernetes.io/projected/19e79f7e-972d-4b12-8b33-ff4108d9bd37-kube-api-access-88hws\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940444    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:50.940609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:50.940453    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f2c433f-eed8-48c1-b2aa-e46df5b91887-bundle\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:37:51.493268 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.493231    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b" event={"ID":"9fc3beff-53a6-42d4-b8d9-54acf279ab51","Type":"ContainerDied","Data":"07e7084aa091154d17681b7cb020586fd6bf46b1c7f64425b4fa7ff924c443b4"}
Oct 30 12:37:51.493676 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.493245    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lt96b"
Oct 30 12:37:51.493676 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.493326    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07e7084aa091154d17681b7cb020586fd6bf46b1c7f64425b4fa7ff924c443b4"
Oct 30 12:37:51.495144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.495123    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx"
Oct 30 12:37:51.495282 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.495126    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c529wx" event={"ID":"5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376","Type":"ContainerDied","Data":"c45eb739ff8612c5284820b5fb8161a5921b3b01abe4badfefcd1cfb4d356365"}
Oct 30 12:37:51.495282 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.495244    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45eb739ff8612c5284820b5fb8161a5921b3b01abe4badfefcd1cfb4d356365"
Oct 30 12:37:51.497148 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.497121    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x" event={"ID":"19e79f7e-972d-4b12-8b33-ff4108d9bd37","Type":"ContainerDied","Data":"263c8a8e757da3492151455fb5ec188442fc4484a412dee57d5cdcb671b55179"}
Oct 30 12:37:51.497266 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.497161    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="263c8a8e757da3492151455fb5ec188442fc4484a412dee57d5cdcb671b55179"
Oct 30 12:37:51.497266 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.497128    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvcl9x"
Oct 30 12:37:51.499243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.499221    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk"
Oct 30 12:37:51.499348 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.499233    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rtnk" event={"ID":"2f2c433f-eed8-48c1-b2aa-e46df5b91887","Type":"ContainerDied","Data":"5931c3dbccc85c4957536343c396b47868b990c83dcdee82127eb6b8daf839b2"}
Oct 30 12:37:51.499348 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.499277    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5931c3dbccc85c4957536343c396b47868b990c83dcdee82127eb6b8daf839b2"
Oct 30 12:37:51.501366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.501332    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc" event={"ID":"71de8d5d-ef64-47cc-a616-604fba27de01","Type":"ContainerDied","Data":"2cf455f10b2285929a617939682bed1aaf1f10e4563f695e4d2677754d705e18"}
Oct 30 12:37:51.501366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.501353    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvrtc"
Oct 30 12:37:51.501527 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.501376    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf455f10b2285929a617939682bed1aaf1f10e4563f695e4d2677754d705e18"
Oct 30 12:37:51.706073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706035    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"]
Oct 30 12:37:51.706374 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706361    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerName="extract"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706376    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerName="extract"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706387    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerName="pull"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706392    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerName="pull"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706401    4993 cpu_manager.go:414] "containerMap: removing container" podUID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerName="pull"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706406    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerName="pull"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706413    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2827150c-2b55-41b4-a73e-b20450665e15" containerName="extract"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706417    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2827150c-2b55-41b4-a73e-b20450665e15" containerName="extract"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706426    4993 cpu_manager.go:414] "containerMap: removing container" podUID="71de8d5d-ef64-47cc-a616-604fba27de01" containerName="util"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706430    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71de8d5d-ef64-47cc-a616-604fba27de01" containerName="util"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706436    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerName="extract"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706441    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerName="extract"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706447    4993 cpu_manager.go:414] "containerMap: removing container" podUID="71de8d5d-ef64-47cc-a616-604fba27de01" containerName="extract"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706451    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71de8d5d-ef64-47cc-a616-604fba27de01" containerName="extract"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706461    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerName="util"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706465    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerName="util"
Oct 30 12:37:51.706466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706472    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706476    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706482    4993 cpu_manager.go:414] "containerMap: removing container" podUID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706486    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706492    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706498    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706508    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2827150c-2b55-41b4-a73e-b20450665e15" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706512    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2827150c-2b55-41b4-a73e-b20450665e15" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706518    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706522    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706530    4993 cpu_manager.go:414] "containerMap: removing container" podUID="71de8d5d-ef64-47cc-a616-604fba27de01" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706533    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71de8d5d-ef64-47cc-a616-604fba27de01" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706543    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2827150c-2b55-41b4-a73e-b20450665e15" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706547    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2827150c-2b55-41b4-a73e-b20450665e15" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706557    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706561    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706566    4993 cpu_manager.go:414] "containerMap: removing container" podUID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706570    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706577    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706582    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerName="pull"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706588    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706592    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706599    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706603    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerName="util"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706609    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706612    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706655    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3796532e-93ff-4a27-9c23-187bcb25b9d5" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706663    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1ddcdc-cf61-4c62-bce2-36b0bc4f3376" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706668    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc3beff-53a6-42d4-b8d9-54acf279ab51" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706673    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2827150c-2b55-41b4-a73e-b20450665e15" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706678    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2c433f-eed8-48c1-b2aa-e46df5b91887" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706683    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="71de8d5d-ef64-47cc-a616-604fba27de01" containerName="extract"
Oct 30 12:37:51.706900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.706692    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e79f7e-972d-4b12-8b33-ff4108d9bd37" containerName="extract"
Oct 30 12:37:51.709948 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.709930    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.738513 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.738479    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 12:37:51.738710 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.738583    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 12:37:51.738710 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.738640    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 12:37:51.738710 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.738704    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n948z\""
Oct 30 12:37:51.746055 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.745995    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b202f0f-9934-474b-a3b2-ebb6805e37ee-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.746055 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.746034    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.746163 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.746149    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6gvb\" (UniqueName: \"kubernetes.io/projected/0b202f0f-9934-474b-a3b2-ebb6805e37ee-kube-api-access-g6gvb\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.746265 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.746247    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.746320 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.746293    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b202f0f-9934-474b-a3b2-ebb6805e37ee-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.746356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.746319    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.754385 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.754363    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 12:37:51.754895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.754879    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 12:37:51.782965 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.782930    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"]
Oct 30 12:37:51.847766 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.847726    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6gvb\" (UniqueName: \"kubernetes.io/projected/0b202f0f-9934-474b-a3b2-ebb6805e37ee-kube-api-access-g6gvb\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.847766 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.847770    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.847988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.847798    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b202f0f-9934-474b-a3b2-ebb6805e37ee-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.847988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.847819    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.847988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.847843    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b202f0f-9934-474b-a3b2-ebb6805e37ee-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.847988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.847871    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.848355 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.848332    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b202f0f-9934-474b-a3b2-ebb6805e37ee-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.850334 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.850302    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b202f0f-9934-474b-a3b2-ebb6805e37ee-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.850532 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.850513    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.850591 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.850569    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.850658 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.850594    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b202f0f-9934-474b-a3b2-ebb6805e37ee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:51.877292 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:51.877254    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6gvb\" (UniqueName: \"kubernetes.io/projected/0b202f0f-9934-474b-a3b2-ebb6805e37ee-kube-api-access-g6gvb\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-j4rwb\" (UID: \"0b202f0f-9934-474b-a3b2-ebb6805e37ee\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:52.023879 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:52.023779    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:52.375581 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:52.375547    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"]
Oct 30 12:37:52.376952 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:37:52.376926    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b202f0f_9934_474b_a3b2_ebb6805e37ee.slice/crio-11c7b056a979b6a8bd7bfc50279020ae523ad19dbbefff03188cb7fea7e8f15f WatchSource:0}: Error finding container 11c7b056a979b6a8bd7bfc50279020ae523ad19dbbefff03188cb7fea7e8f15f: Status 404 returned error can't find the container with id 11c7b056a979b6a8bd7bfc50279020ae523ad19dbbefff03188cb7fea7e8f15f
Oct 30 12:37:52.507175 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:52.507134    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb" event={"ID":"0b202f0f-9934-474b-a3b2-ebb6805e37ee","Type":"ContainerStarted","Data":"11c7b056a979b6a8bd7bfc50279020ae523ad19dbbefff03188cb7fea7e8f15f"}
Oct 30 12:37:56.528453 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:56.528409    4993 generic.go:356] "Generic (PLEG): container finished" podID="0b202f0f-9934-474b-a3b2-ebb6805e37ee" containerID="c9d00db21e1a8d2405807ceb42c4a8e617772bb88480bdde4e605b3e8874335f" exitCode=0
Oct 30 12:37:56.528848 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:56.528495    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb" event={"ID":"0b202f0f-9934-474b-a3b2-ebb6805e37ee","Type":"ContainerDied","Data":"c9d00db21e1a8d2405807ceb42c4a8e617772bb88480bdde4e605b3e8874335f"}
Oct 30 12:37:57.534818 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:57.534779    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb" event={"ID":"0b202f0f-9934-474b-a3b2-ebb6805e37ee","Type":"ContainerStarted","Data":"573a8a6d70ee419518c83e208299fc5b81be6bcb83ac67004b13f7187095b163"}
Oct 30 12:37:57.535240 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:57.534961    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:37:57.642710 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:37:57.642655    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb" podStartSLOduration=2.626698236 podStartE2EDuration="6.642638984s" totalImagesPullingTime="3.559104115s" totalInitContainerRuntime="456.836633ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:57.630077278 +0000 UTC m=+342.105796888" watchObservedRunningTime="2026-10-30 12:37:57.642638984 +0000 UTC m=+342.118358593"
Oct 30 12:38:05.958264 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:05.958228    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"]
Oct 30 12:38:05.964981 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:05.964951    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"
Oct 30 12:38:06.012777 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.012741    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 12:38:06.012777 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.012760    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-859bz\""
Oct 30 12:38:06.012777 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.012755    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:38:06.014623 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.014588    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"]
Oct 30 12:38:06.070848 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.070809    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72lw5\" (UniqueName: \"kubernetes.io/projected/75161b9d-7dd7-42b1-b53a-3aea53e0e2c4-kube-api-access-72lw5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bczdg\" (UID: \"75161b9d-7dd7-42b1-b53a-3aea53e0e2c4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"
Oct 30 12:38:06.071058 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.070864    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75161b9d-7dd7-42b1-b53a-3aea53e0e2c4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bczdg\" (UID: \"75161b9d-7dd7-42b1-b53a-3aea53e0e2c4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"
Oct 30 12:38:06.172250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.172177    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75161b9d-7dd7-42b1-b53a-3aea53e0e2c4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bczdg\" (UID: \"75161b9d-7dd7-42b1-b53a-3aea53e0e2c4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"
Oct 30 12:38:06.172460 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.172328    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lw5\" (UniqueName: \"kubernetes.io/projected/75161b9d-7dd7-42b1-b53a-3aea53e0e2c4-kube-api-access-72lw5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bczdg\" (UID: \"75161b9d-7dd7-42b1-b53a-3aea53e0e2c4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"
Oct 30 12:38:06.172578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.172555    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75161b9d-7dd7-42b1-b53a-3aea53e0e2c4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bczdg\" (UID: \"75161b9d-7dd7-42b1-b53a-3aea53e0e2c4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"
Oct 30 12:38:06.203515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.203477    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lw5\" (UniqueName: \"kubernetes.io/projected/75161b9d-7dd7-42b1-b53a-3aea53e0e2c4-kube-api-access-72lw5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bczdg\" (UID: \"75161b9d-7dd7-42b1-b53a-3aea53e0e2c4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"
Oct 30 12:38:06.278777 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.278734    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"
Oct 30 12:38:06.613020 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:06.612989    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg"]
Oct 30 12:38:06.614868 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:38:06.614838    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75161b9d_7dd7_42b1_b53a_3aea53e0e2c4.slice/crio-97269dbef5313d80ba9e62288b77401c970f6eedbfd98ea7316fc089691d6b57 WatchSource:0}: Error finding container 97269dbef5313d80ba9e62288b77401c970f6eedbfd98ea7316fc089691d6b57: Status 404 returned error can't find the container with id 97269dbef5313d80ba9e62288b77401c970f6eedbfd98ea7316fc089691d6b57
Oct 30 12:38:07.581181 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:07.581133    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg" event={"ID":"75161b9d-7dd7-42b1-b53a-3aea53e0e2c4","Type":"ContainerStarted","Data":"97269dbef5313d80ba9e62288b77401c970f6eedbfd98ea7316fc089691d6b57"}
Oct 30 12:38:08.542231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:08.542187    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-j4rwb"
Oct 30 12:38:08.569743 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:38:08.569703    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:38:08.588643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:08.588604    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg" event={"ID":"75161b9d-7dd7-42b1-b53a-3aea53e0e2c4","Type":"ContainerStarted","Data":"4d13189a280a81c5db1d2ae85f6c93d3d9df02ac9cd26e945dbe32827835bec5"}
Oct 30 12:38:08.687463 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:08.687400    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bczdg" podStartSLOduration=2.305260474 podStartE2EDuration="3.687381233s" totalImagesPullingTime="1.382120759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:08.685765027 +0000 UTC m=+353.161484637" watchObservedRunningTime="2026-10-30 12:38:08.687381233 +0000 UTC m=+353.163100840"
Oct 30 12:38:11.258612 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.258575    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"]
Oct 30 12:38:11.262067 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.262044    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"
Oct 30 12:38:11.313971 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.313929    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 12:38:11.313971 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.313929    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 12:38:11.317998 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.317969    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-777wn\" (UniqueName: \"kubernetes.io/projected/9508df2d-3fcd-41aa-8559-2def2056346c-kube-api-access-777wn\") pod \"limitador-operator-controller-manager-5778bb4449-tvfdn\" (UID: \"9508df2d-3fcd-41aa-8559-2def2056346c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"
Oct 30 12:38:11.320539 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.320509    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"]
Oct 30 12:38:11.359557 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.359521    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kx7xp\""
Oct 30 12:38:11.419087 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.419048    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-777wn\" (UniqueName: \"kubernetes.io/projected/9508df2d-3fcd-41aa-8559-2def2056346c-kube-api-access-777wn\") pod \"limitador-operator-controller-manager-5778bb4449-tvfdn\" (UID: \"9508df2d-3fcd-41aa-8559-2def2056346c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"
Oct 30 12:38:11.451096 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.451061    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-777wn\" (UniqueName: \"kubernetes.io/projected/9508df2d-3fcd-41aa-8559-2def2056346c-kube-api-access-777wn\") pod \"limitador-operator-controller-manager-5778bb4449-tvfdn\" (UID: \"9508df2d-3fcd-41aa-8559-2def2056346c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"
Oct 30 12:38:11.574951 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.574850    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"
Oct 30 12:38:11.893384 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:11.893352    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"]
Oct 30 12:38:11.895464 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:38:11.895430    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9508df2d_3fcd_41aa_8559_2def2056346c.slice/crio-018b9cb04ff0ca1108b2a5c178e12986fedcf561ae265b44594f44189e550241 WatchSource:0}: Error finding container 018b9cb04ff0ca1108b2a5c178e12986fedcf561ae265b44594f44189e550241: Status 404 returned error can't find the container with id 018b9cb04ff0ca1108b2a5c178e12986fedcf561ae265b44594f44189e550241
Oct 30 12:38:12.613858 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:12.613811    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn" event={"ID":"9508df2d-3fcd-41aa-8559-2def2056346c","Type":"ContainerStarted","Data":"018b9cb04ff0ca1108b2a5c178e12986fedcf561ae265b44594f44189e550241"}
Oct 30 12:38:14.624639 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:14.624599    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn" event={"ID":"9508df2d-3fcd-41aa-8559-2def2056346c","Type":"ContainerStarted","Data":"c0fe2f73985c4afef68afeb32cfa532e4b917d570028568ed3953ea21318f29b"}
Oct 30 12:38:14.625143 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:14.624710    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"
Oct 30 12:38:14.684680 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:14.684614    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn" podStartSLOduration=1.566657105 podStartE2EDuration="3.684599661s" totalImagesPullingTime="2.117942556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:14.681699462 +0000 UTC m=+359.157419071" watchObservedRunningTime="2026-10-30 12:38:14.684599661 +0000 UTC m=+359.160319351"
Oct 30 12:38:19.073937 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.073899    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"]
Oct 30 12:38:19.078840 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.078818    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:19.097131 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.097100    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r4h5x\""
Oct 30 12:38:19.110351 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.110316    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"]
Oct 30 12:38:19.190276 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.190196    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f5b18c-669d-45d7-a435-cc2028580edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fpw7s\" (UID: \"85f5b18c-669d-45d7-a435-cc2028580edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:19.190475 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.190351    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr899\" (UniqueName: \"kubernetes.io/projected/85f5b18c-669d-45d7-a435-cc2028580edf-kube-api-access-lr899\") pod \"kuadrant-operator-controller-manager-678d87766f-fpw7s\" (UID: \"85f5b18c-669d-45d7-a435-cc2028580edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:19.291589 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.291545    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f5b18c-669d-45d7-a435-cc2028580edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fpw7s\" (UID: \"85f5b18c-669d-45d7-a435-cc2028580edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:19.291771 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.291600    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr899\" (UniqueName: \"kubernetes.io/projected/85f5b18c-669d-45d7-a435-cc2028580edf-kube-api-access-lr899\") pod \"kuadrant-operator-controller-manager-678d87766f-fpw7s\" (UID: \"85f5b18c-669d-45d7-a435-cc2028580edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:19.291967 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.291943    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f5b18c-669d-45d7-a435-cc2028580edf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fpw7s\" (UID: \"85f5b18c-669d-45d7-a435-cc2028580edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:19.323024 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.322984    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr899\" (UniqueName: \"kubernetes.io/projected/85f5b18c-669d-45d7-a435-cc2028580edf-kube-api-access-lr899\") pod \"kuadrant-operator-controller-manager-678d87766f-fpw7s\" (UID: \"85f5b18c-669d-45d7-a435-cc2028580edf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:19.392132 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.392023    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:19.726688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:19.726658    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"]
Oct 30 12:38:19.728326 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:38:19.728283    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f5b18c_669d_45d7_a435_cc2028580edf.slice/crio-e356b6af24c1cf04f95ce9371a8f4079115c087d09e16d5fc434da7f9217202e WatchSource:0}: Error finding container e356b6af24c1cf04f95ce9371a8f4079115c087d09e16d5fc434da7f9217202e: Status 404 returned error can't find the container with id e356b6af24c1cf04f95ce9371a8f4079115c087d09e16d5fc434da7f9217202e
Oct 30 12:38:20.657515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:20.657468    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s" event={"ID":"85f5b18c-669d-45d7-a435-cc2028580edf","Type":"ContainerStarted","Data":"e356b6af24c1cf04f95ce9371a8f4079115c087d09e16d5fc434da7f9217202e"}
Oct 30 12:38:23.677233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:23.677123    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s" event={"ID":"85f5b18c-669d-45d7-a435-cc2028580edf","Type":"ContainerStarted","Data":"9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4"}
Oct 30 12:38:23.677661 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:23.677286    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:23.747277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:23.747197    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s" podStartSLOduration=1.207785463 podStartE2EDuration="4.74718224s" totalImagesPullingTime="3.539396777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:23.745510789 +0000 UTC m=+368.221230399" watchObservedRunningTime="2026-10-30 12:38:23.74718224 +0000 UTC m=+368.222901849"
Oct 30 12:38:23.787121 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:23.787079    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"]
Oct 30 12:38:23.790729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:23.790706    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"
Oct 30 12:38:23.815944 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:23.815905    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dnhdd\""
Oct 30 12:38:23.817383 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:23.817351    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"]
Oct 30 12:38:23.931589 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:23.931476    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q827j\" (UniqueName: \"kubernetes.io/projected/8b08b699-9669-481c-9622-e82c35b0e729-kube-api-access-q827j\") pod \"authorino-operator-7b7bb4bc77-7vzgf\" (UID: \"8b08b699-9669-481c-9622-e82c35b0e729\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"
Oct 30 12:38:24.032933 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:24.032890    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q827j\" (UniqueName: \"kubernetes.io/projected/8b08b699-9669-481c-9622-e82c35b0e729-kube-api-access-q827j\") pod \"authorino-operator-7b7bb4bc77-7vzgf\" (UID: \"8b08b699-9669-481c-9622-e82c35b0e729\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"
Oct 30 12:38:24.064849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:24.064817    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q827j\" (UniqueName: \"kubernetes.io/projected/8b08b699-9669-481c-9622-e82c35b0e729-kube-api-access-q827j\") pod \"authorino-operator-7b7bb4bc77-7vzgf\" (UID: \"8b08b699-9669-481c-9622-e82c35b0e729\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"
Oct 30 12:38:24.105406 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:24.105360    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"
Oct 30 12:38:24.404373 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:24.404345    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"]
Oct 30 12:38:24.407060 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:38:24.407027    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b08b699_9669_481c_9622_e82c35b0e729.slice/crio-0c5baaf929829a9534e5abd3463eb5107f1bc93febf4b2869eba488133d1572f WatchSource:0}: Error finding container 0c5baaf929829a9534e5abd3463eb5107f1bc93febf4b2869eba488133d1572f: Status 404 returned error can't find the container with id 0c5baaf929829a9534e5abd3463eb5107f1bc93febf4b2869eba488133d1572f
Oct 30 12:38:24.683201 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:24.683111    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf" event={"ID":"8b08b699-9669-481c-9622-e82c35b0e729","Type":"ContainerStarted","Data":"0c5baaf929829a9534e5abd3463eb5107f1bc93febf4b2869eba488133d1572f"}
Oct 30 12:38:25.312106 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.312057    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"]
Oct 30 12:38:25.317450 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.317412    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.344274 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.344223    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dwtcf\""
Oct 30 12:38:25.345602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.345538    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"]
Oct 30 12:38:25.444741 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.444695    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.444741 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.444737    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.444961 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.444760    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38996bb0-1dd5-449a-b542-814106c08ff6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.444961 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.444899    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38996bb0-1dd5-449a-b542-814106c08ff6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.444961 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.444952    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.445073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.445028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.445073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.445053    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qgv\" (UniqueName: \"kubernetes.io/projected/38996bb0-1dd5-449a-b542-814106c08ff6-kube-api-access-v2qgv\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.445143 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.445080    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.445143 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.445098    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38996bb0-1dd5-449a-b542-814106c08ff6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546581 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546553    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546592    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546613    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38996bb0-1dd5-449a-b542-814106c08ff6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546652    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38996bb0-1dd5-449a-b542-814106c08ff6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546695    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546730    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546756    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qgv\" (UniqueName: \"kubernetes.io/projected/38996bb0-1dd5-449a-b542-814106c08ff6-kube-api-access-v2qgv\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546790    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.546827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.546814    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38996bb0-1dd5-449a-b542-814106c08ff6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.547109 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.547046    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.547257 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.547105    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.547497 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.547475    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.547602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.547572    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.547756 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.547737    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38996bb0-1dd5-449a-b542-814106c08ff6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.549587 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.549564    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38996bb0-1dd5-449a-b542-814106c08ff6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.550148 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.550121    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38996bb0-1dd5-449a-b542-814106c08ff6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.574120 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.574040    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38996bb0-1dd5-449a-b542-814106c08ff6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.574120 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.574060    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qgv\" (UniqueName: \"kubernetes.io/projected/38996bb0-1dd5-449a-b542-814106c08ff6-kube-api-access-v2qgv\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5mf69c\" (UID: \"38996bb0-1dd5-449a-b542-814106c08ff6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.631887 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.631850    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tvfdn"
Oct 30 12:38:25.635524 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.635489    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:25.712599 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.712558    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf" event={"ID":"8b08b699-9669-481c-9622-e82c35b0e729","Type":"ContainerStarted","Data":"3d3358999572c8f7bf6c20270cd4d5b496a1ea257019066427695fffb3117113"}
Oct 30 12:38:25.713323 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.713304    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"
Oct 30 12:38:25.760343 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:25.759827    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf" podStartSLOduration=1.691138083 podStartE2EDuration="2.759808094s" totalImagesPullingTime="1.068670011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:25.757277551 +0000 UTC m=+370.232997162" watchObservedRunningTime="2026-10-30 12:38:25.759808094 +0000 UTC m=+370.235527705"
Oct 30 12:38:26.032191 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:26.032161    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"]
Oct 30 12:38:26.034560 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:38:26.034526    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38996bb0_1dd5_449a_b542_814106c08ff6.slice/crio-594af47f60acaca72096192e8c571c77584cb8f1ed57424a73ba708d40e1fe8d WatchSource:0}: Error finding container 594af47f60acaca72096192e8c571c77584cb8f1ed57424a73ba708d40e1fe8d: Status 404 returned error can't find the container with id 594af47f60acaca72096192e8c571c77584cb8f1ed57424a73ba708d40e1fe8d
Oct 30 12:38:26.036761 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:26.036730    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:38:26.036826 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:26.036797    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:38:26.036863 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:26.036850    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:38:26.721569 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:26.721527    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c" event={"ID":"38996bb0-1dd5-449a-b542-814106c08ff6","Type":"ContainerStarted","Data":"8e03b2972b99d68814a7e3b868462c57dd3d012f2bc137e8d242ab1d7dbd753d"}
Oct 30 12:38:26.721997 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:26.721605    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c" event={"ID":"38996bb0-1dd5-449a-b542-814106c08ff6","Type":"ContainerStarted","Data":"594af47f60acaca72096192e8c571c77584cb8f1ed57424a73ba708d40e1fe8d"}
Oct 30 12:38:26.787008 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:26.786947    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c" podStartSLOduration=1.786930745 podStartE2EDuration="1.786930745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:26.785910957 +0000 UTC m=+371.261630568" watchObservedRunningTime="2026-10-30 12:38:26.786930745 +0000 UTC m=+371.262650402"
Oct 30 12:38:27.635859 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:27.635805    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:27.640968 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:27.640937    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:27.726483 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:27.726450    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:27.727560 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:27.727541    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5mf69c"
Oct 30 12:38:34.685555 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:34.685518    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:38:37.728944 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:38:37.728906    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-7vzgf"
Oct 30 12:38:38.568658 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:38:38.568582    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:39:08.568951 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:39:08.568865    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:39:25.780467 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.780432    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mp6wv"]
Oct 30 12:39:25.783068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.783045    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:25.807338 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.807290    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mp6wv"]
Oct 30 12:39:25.817828 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.817794    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 12:39:25.821373 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.821341    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 12:39:25.825894 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.825863    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 12:39:25.826065 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.825899    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-swklv\""
Oct 30 12:39:25.852144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.852108    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"]
Oct 30 12:39:25.854776 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.854756    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:25.886027 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.885996    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831c60d3-728e-4e13-8e6b-4b42d922c47e-cert\") pod \"kserve-controller-manager-776cc66d47-mp6wv\" (UID: \"831c60d3-728e-4e13-8e6b-4b42d922c47e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:25.886027 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.886033    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgjrd\" (UniqueName: \"kubernetes.io/projected/831c60d3-728e-4e13-8e6b-4b42d922c47e-kube-api-access-dgjrd\") pod \"kserve-controller-manager-776cc66d47-mp6wv\" (UID: \"831c60d3-728e-4e13-8e6b-4b42d922c47e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:25.904976 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.904938    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"]
Oct 30 12:39:25.924994 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.924963    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 12:39:25.924994 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.924964    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-qzllf\""
Oct 30 12:39:25.981099 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.981060    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bt6xn"]
Oct 30 12:39:25.983455 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.983434    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:25.986605 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.986575    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831c60d3-728e-4e13-8e6b-4b42d922c47e-cert\") pod \"kserve-controller-manager-776cc66d47-mp6wv\" (UID: \"831c60d3-728e-4e13-8e6b-4b42d922c47e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:25.986745 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.986611    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgjrd\" (UniqueName: \"kubernetes.io/projected/831c60d3-728e-4e13-8e6b-4b42d922c47e-kube-api-access-dgjrd\") pod \"kserve-controller-manager-776cc66d47-mp6wv\" (UID: \"831c60d3-728e-4e13-8e6b-4b42d922c47e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:25.986745 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.986652    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db5d16-3638-4b85-ae44-69fb57b3c77a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8mvtq\" (UID: \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:25.986745 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.986713    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frpmw\" (UniqueName: \"kubernetes.io/projected/a6db5d16-3638-4b85-ae44-69fb57b3c77a-kube-api-access-frpmw\") pod \"llmisvc-controller-manager-869d5dcc5d-8mvtq\" (UID: \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:25.989327 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:25.989302    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831c60d3-728e-4e13-8e6b-4b42d922c47e-cert\") pod \"kserve-controller-manager-776cc66d47-mp6wv\" (UID: \"831c60d3-728e-4e13-8e6b-4b42d922c47e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:26.001555 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.001524    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-zm28s"]
Oct 30 12:39:26.003918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.003893    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.008947 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.008914    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bt6xn"]
Oct 30 12:39:26.012938 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.012910    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 12:39:26.013097 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.012953    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-q6kmq\""
Oct 30 12:39:26.019184 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.019147    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgjrd\" (UniqueName: \"kubernetes.io/projected/831c60d3-728e-4e13-8e6b-4b42d922c47e-kube-api-access-dgjrd\") pod \"kserve-controller-manager-776cc66d47-mp6wv\" (UID: \"831c60d3-728e-4e13-8e6b-4b42d922c47e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:26.032158 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.032073    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zm28s"]
Oct 30 12:39:26.087202 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.087156    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700e35bb-15f5-4740-8132-fa48da68e589-tls-certs\") pod \"model-serving-api-75bd89fdfc-bt6xn\" (UID: \"700e35bb-15f5-4740-8132-fa48da68e589\") " pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:26.087202 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.087221    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db5d16-3638-4b85-ae44-69fb57b3c77a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8mvtq\" (UID: \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:26.087469 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.087336    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7plk8\" (UniqueName: \"kubernetes.io/projected/700e35bb-15f5-4740-8132-fa48da68e589-kube-api-access-7plk8\") pod \"model-serving-api-75bd89fdfc-bt6xn\" (UID: \"700e35bb-15f5-4740-8132-fa48da68e589\") " pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:26.087469 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.087397    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frpmw\" (UniqueName: \"kubernetes.io/projected/a6db5d16-3638-4b85-ae44-69fb57b3c77a-kube-api-access-frpmw\") pod \"llmisvc-controller-manager-869d5dcc5d-8mvtq\" (UID: \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:26.089772 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.089748    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db5d16-3638-4b85-ae44-69fb57b3c77a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8mvtq\" (UID: \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:26.095678 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.095644    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:26.111022 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.110980    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frpmw\" (UniqueName: \"kubernetes.io/projected/a6db5d16-3638-4b85-ae44-69fb57b3c77a-kube-api-access-frpmw\") pod \"llmisvc-controller-manager-869d5dcc5d-8mvtq\" (UID: \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:26.111489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.111466    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6x922\""
Oct 30 12:39:26.111639 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.111610    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 12:39:26.111743 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.111721    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 12:39:26.165823 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.165787    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:26.188917 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.188877    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpfn\" (UniqueName: \"kubernetes.io/projected/85eadb46-be52-4e6d-8813-46bb7b155a39-kube-api-access-khpfn\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.189075 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.188938    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700e35bb-15f5-4740-8132-fa48da68e589-tls-certs\") pod \"model-serving-api-75bd89fdfc-bt6xn\" (UID: \"700e35bb-15f5-4740-8132-fa48da68e589\") " pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:26.189075 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.189060    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7plk8\" (UniqueName: \"kubernetes.io/projected/700e35bb-15f5-4740-8132-fa48da68e589-kube-api-access-7plk8\") pod \"model-serving-api-75bd89fdfc-bt6xn\" (UID: \"700e35bb-15f5-4740-8132-fa48da68e589\") " pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:26.189195 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.189124    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85eadb46-be52-4e6d-8813-46bb7b155a39-cert\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.189195 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.189148    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85eadb46-be52-4e6d-8813-46bb7b155a39-metrics-certs\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.191911 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.191879    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700e35bb-15f5-4740-8132-fa48da68e589-tls-certs\") pod \"model-serving-api-75bd89fdfc-bt6xn\" (UID: \"700e35bb-15f5-4740-8132-fa48da68e589\") " pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:26.212900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.212717    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7plk8\" (UniqueName: \"kubernetes.io/projected/700e35bb-15f5-4740-8132-fa48da68e589-kube-api-access-7plk8\") pod \"model-serving-api-75bd89fdfc-bt6xn\" (UID: \"700e35bb-15f5-4740-8132-fa48da68e589\") " pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:26.290493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.290402    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85eadb46-be52-4e6d-8813-46bb7b155a39-cert\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.290493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.290449    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85eadb46-be52-4e6d-8813-46bb7b155a39-metrics-certs\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.290493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.290492    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpfn\" (UniqueName: \"kubernetes.io/projected/85eadb46-be52-4e6d-8813-46bb7b155a39-kube-api-access-khpfn\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.293142 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.293104    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85eadb46-be52-4e6d-8813-46bb7b155a39-metrics-certs\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.293345 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.293153    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85eadb46-be52-4e6d-8813-46bb7b155a39-cert\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.304817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.304788    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:26.313709 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.313680    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpfn\" (UniqueName: \"kubernetes.io/projected/85eadb46-be52-4e6d-8813-46bb7b155a39-kube-api-access-khpfn\") pod \"odh-model-controller-769875db56-zm28s\" (UID: \"85eadb46-be52-4e6d-8813-46bb7b155a39\") " pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.319523 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.319495    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:26.393194 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.393166    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mp6wv"]
Oct 30 12:39:26.465926 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.465898    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"]
Oct 30 12:39:26.467674 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:39:26.467638    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda6db5d16_3638_4b85_ae44_69fb57b3c77a.slice/crio-aed36845d7fbaa53c454ba44cf44bf8a4dc367c5bd9315ab10d4e193c4e59f6e WatchSource:0}: Error finding container aed36845d7fbaa53c454ba44cf44bf8a4dc367c5bd9315ab10d4e193c4e59f6e: Status 404 returned error can't find the container with id aed36845d7fbaa53c454ba44cf44bf8a4dc367c5bd9315ab10d4e193c4e59f6e
Oct 30 12:39:26.605949 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.605920    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bt6xn"]
Oct 30 12:39:26.608568 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:39:26.608537    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700e35bb_15f5_4740_8132_fa48da68e589.slice/crio-92c7e5263fcb47c4ec1d5e5fea9d2a7419f41c6ac00580c000dee07d1aba9e2e WatchSource:0}: Error finding container 92c7e5263fcb47c4ec1d5e5fea9d2a7419f41c6ac00580c000dee07d1aba9e2e: Status 404 returned error can't find the container with id 92c7e5263fcb47c4ec1d5e5fea9d2a7419f41c6ac00580c000dee07d1aba9e2e
Oct 30 12:39:26.622896 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:26.622866    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zm28s"]
Oct 30 12:39:26.624932 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:39:26.624895    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85eadb46_be52_4e6d_8813_46bb7b155a39.slice/crio-56e9f07a5327dc6328cdefd2510198b05ebdde4783d5301a47b4c1efcfe893c1 WatchSource:0}: Error finding container 56e9f07a5327dc6328cdefd2510198b05ebdde4783d5301a47b4c1efcfe893c1: Status 404 returned error can't find the container with id 56e9f07a5327dc6328cdefd2510198b05ebdde4783d5301a47b4c1efcfe893c1
Oct 30 12:39:27.063251 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:27.055849    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv" event={"ID":"831c60d3-728e-4e13-8e6b-4b42d922c47e","Type":"ContainerStarted","Data":"5cd0244ffb980bc1441e9991eda91c40bf92260dbe5157ccecbe2c0b36144c9d"}
Oct 30 12:39:27.063251 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:27.059201    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zm28s" event={"ID":"85eadb46-be52-4e6d-8813-46bb7b155a39","Type":"ContainerStarted","Data":"56e9f07a5327dc6328cdefd2510198b05ebdde4783d5301a47b4c1efcfe893c1"}
Oct 30 12:39:27.067768 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:27.067725    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn" event={"ID":"700e35bb-15f5-4740-8132-fa48da68e589","Type":"ContainerStarted","Data":"92c7e5263fcb47c4ec1d5e5fea9d2a7419f41c6ac00580c000dee07d1aba9e2e"}
Oct 30 12:39:27.073400 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:27.073338    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq" event={"ID":"a6db5d16-3638-4b85-ae44-69fb57b3c77a","Type":"ContainerStarted","Data":"aed36845d7fbaa53c454ba44cf44bf8a4dc367c5bd9315ab10d4e193c4e59f6e"}
Oct 30 12:39:30.100397 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:30.100352    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn" event={"ID":"700e35bb-15f5-4740-8132-fa48da68e589","Type":"ContainerStarted","Data":"ea0bd60236ab6506e287c3a5c22f8ab1da8ad6f7f46802ce7105afffba7fd384"}
Oct 30 12:39:30.100900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:30.100486    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:30.142144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:30.141648    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn" podStartSLOduration=2.14792362 podStartE2EDuration="5.14162494s" totalImagesPullingTime="2.99370132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:30.141355657 +0000 UTC m=+434.617075268" watchObservedRunningTime="2026-10-30 12:39:30.14162494 +0000 UTC m=+434.617344562"
Oct 30 12:39:31.108888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:31.108768    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zm28s" event={"ID":"85eadb46-be52-4e6d-8813-46bb7b155a39","Type":"ContainerStarted","Data":"4f142058adb1de41ecdc681872c486d79419ec2c7d6641927f5291ed88ff75fc"}
Oct 30 12:39:31.108888 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:31.108884    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:39:31.110701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:31.110676    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq" event={"ID":"a6db5d16-3638-4b85-ae44-69fb57b3c77a","Type":"ContainerStarted","Data":"03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5"}
Oct 30 12:39:31.110817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:31.110729    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:39:31.112360 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:31.112330    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv" event={"ID":"831c60d3-728e-4e13-8e6b-4b42d922c47e","Type":"ContainerStarted","Data":"cdf410055fa6193cb41bbda6189934fbb959fdc3628ae152f6efa964b40f7d72"}
Oct 30 12:39:31.238749 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:31.238688    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv" podStartSLOduration=1.879806666 podStartE2EDuration="6.23867278s" totalImagesPullingTime="4.358866114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:31.238219113 +0000 UTC m=+435.713938715" watchObservedRunningTime="2026-10-30 12:39:31.23867278 +0000 UTC m=+435.714392390"
Oct 30 12:39:31.239567 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:31.239521    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-zm28s" podStartSLOduration=2.320305817 podStartE2EDuration="6.23950688s" totalImagesPullingTime="3.919201063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:31.188297455 +0000 UTC m=+435.664017065" watchObservedRunningTime="2026-10-30 12:39:31.23950688 +0000 UTC m=+435.715226490"
Oct 30 12:39:31.282703 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:31.282643    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq" podStartSLOduration=1.980229521 podStartE2EDuration="6.282626956s" totalImagesPullingTime="4.302397435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:31.280530474 +0000 UTC m=+435.756250084" watchObservedRunningTime="2026-10-30 12:39:31.282626956 +0000 UTC m=+435.758346563"
Oct 30 12:39:32.117804 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:32.117768    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:39:38.567937 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:39:38.567895    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:39:41.117099 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:41.117062    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-bt6xn"
Oct 30 12:39:42.120526 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:39:42.120485    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-zm28s"
Oct 30 12:40:02.120924 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:40:02.120886    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:40:03.129509 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:40:03.129473    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mp6wv"
Oct 30 12:40:08.569779 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:40:08.569736    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:40:38.567492 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:40:38.567453    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:41:08.568181 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:41:08.568139    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:41:38.567727 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:41:38.567685    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:42:08.570619 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:42:08.570531    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:42:15.742388 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:42:15.742357    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:42:15.743607 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:42:15.743585    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:42:38.567222 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:42:38.567166    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:43:08.568641 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:43:08.568595    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:43:38.568252 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:43:38.568198    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:44:08.569013 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:44:08.568966    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:44:38.568170 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:44:38.568123    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:45:08.567554 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:45:08.567462    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:45:38.567193 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:45:38.567153    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:46:08.569381 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:46:08.569343    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:46:38.568626 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:46:38.568590    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:47:08.569414 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:47:08.569373    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:47:15.781058 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:47:15.781032    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:47:15.784279 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:47:15.784252    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:47:38.566774 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:47:38.566733    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:48:08.568890 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:48:08.568792    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:48:38.567255 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:48:38.567215    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:48:45.420317 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.420278    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fpw7s_85f5b18c-669d-45d7-a435-cc2028580edf/manager/0.log"
Oct 30 12:48:45.676634 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.676549    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"]
Oct 30 12:48:45.676822 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.676777    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s" podUID="85f5b18c-669d-45d7-a435-cc2028580edf" containerName="manager" containerID="cri-o://9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4" gracePeriod=10
Oct 30 12:48:45.695720 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.695684    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"]
Oct 30 12:48:45.699623 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.699594    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:45.710524 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:48:45.710482    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f5b18c_669d_45d7_a435_cc2028580edf.slice/crio-9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 12:48:45.737603 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.737554    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aff0921-3696-4f79-8380-f4bd418fde32-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7hpfd\" (UID: \"0aff0921-3696-4f79-8380-f4bd418fde32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:45.737812 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.737722    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bw2\" (UniqueName: \"kubernetes.io/projected/0aff0921-3696-4f79-8380-f4bd418fde32-kube-api-access-94bw2\") pod \"kuadrant-operator-controller-manager-678d87766f-7hpfd\" (UID: \"0aff0921-3696-4f79-8380-f4bd418fde32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:45.744187 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.743997    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"]
Oct 30 12:48:45.839364 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.839322    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aff0921-3696-4f79-8380-f4bd418fde32-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7hpfd\" (UID: \"0aff0921-3696-4f79-8380-f4bd418fde32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:45.839537 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.839410    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bw2\" (UniqueName: \"kubernetes.io/projected/0aff0921-3696-4f79-8380-f4bd418fde32-kube-api-access-94bw2\") pod \"kuadrant-operator-controller-manager-678d87766f-7hpfd\" (UID: \"0aff0921-3696-4f79-8380-f4bd418fde32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:45.839852 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.839829    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aff0921-3696-4f79-8380-f4bd418fde32-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7hpfd\" (UID: \"0aff0921-3696-4f79-8380-f4bd418fde32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:45.867903 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.867869    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bw2\" (UniqueName: \"kubernetes.io/projected/0aff0921-3696-4f79-8380-f4bd418fde32-kube-api-access-94bw2\") pod \"kuadrant-operator-controller-manager-678d87766f-7hpfd\" (UID: \"0aff0921-3696-4f79-8380-f4bd418fde32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:45.936584 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.936512    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:48:45.964285 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.964242    4993 generic.go:356] "Generic (PLEG): container finished" podID="85f5b18c-669d-45d7-a435-cc2028580edf" containerID="9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4" exitCode=0
Oct 30 12:48:45.964457 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.964292    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"
Oct 30 12:48:45.964457 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.964308    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s" event={"ID":"85f5b18c-669d-45d7-a435-cc2028580edf","Type":"ContainerDied","Data":"9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4"}
Oct 30 12:48:45.964457 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.964361    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s" event={"ID":"85f5b18c-669d-45d7-a435-cc2028580edf","Type":"ContainerDied","Data":"e356b6af24c1cf04f95ce9371a8f4079115c087d09e16d5fc434da7f9217202e"}
Oct 30 12:48:45.964457 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.964386    4993 scope.go:122] "RemoveContainer" containerID="9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4"
Oct 30 12:48:45.973478 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.973455    4993 scope.go:122] "RemoveContainer" containerID="9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4"
Oct 30 12:48:45.973827 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:48:45.973805    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4\": container with ID starting with 9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4 not found: ID does not exist" containerID="9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4"
Oct 30 12:48:45.973875 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:45.973840    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4"} err="failed to get container status \"9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4\": rpc error: code = NotFound desc = could not find container \"9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4\": container with ID starting with 9b7770a961031e6dd7d7bc195def5633959c54c909c63eabe4c144a536f13eb4 not found: ID does not exist"
Oct 30 12:48:46.040476 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.040445    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f5b18c-669d-45d7-a435-cc2028580edf-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f5b18c-669d-45d7-a435-cc2028580edf-extensions-socket-volume\") pod \"85f5b18c-669d-45d7-a435-cc2028580edf\" (UID: \"85f5b18c-669d-45d7-a435-cc2028580edf\") "
Oct 30 12:48:46.040658 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.040526    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f5b18c-669d-45d7-a435-cc2028580edf-kube-api-access-lr899\" (UniqueName: \"kubernetes.io/projected/85f5b18c-669d-45d7-a435-cc2028580edf-kube-api-access-lr899\") pod \"85f5b18c-669d-45d7-a435-cc2028580edf\" (UID: \"85f5b18c-669d-45d7-a435-cc2028580edf\") "
Oct 30 12:48:46.040924 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.040901    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f5b18c-669d-45d7-a435-cc2028580edf-extensions-socket-volume" pod "85f5b18c-669d-45d7-a435-cc2028580edf" (UID: "85f5b18c-669d-45d7-a435-cc2028580edf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:48:46.042746 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.042720    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f5b18c-669d-45d7-a435-cc2028580edf-kube-api-access-lr899" pod "85f5b18c-669d-45d7-a435-cc2028580edf" (UID: "85f5b18c-669d-45d7-a435-cc2028580edf"). InnerVolumeSpecName "kube-api-access-lr899". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:48:46.068981 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.068933    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:46.141643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.141605    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr899\" (UniqueName: \"kubernetes.io/projected/85f5b18c-669d-45d7-a435-cc2028580edf-kube-api-access-lr899\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:48:46.141643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.141645    4993 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f5b18c-669d-45d7-a435-cc2028580edf-extensions-socket-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:48:46.314727 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.314689    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"]
Oct 30 12:48:46.320589 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.320555    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fpw7s"]
Oct 30 12:48:46.393085 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.393059    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"]
Oct 30 12:48:46.973002 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.972961    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd" event={"ID":"0aff0921-3696-4f79-8380-f4bd418fde32","Type":"ContainerStarted","Data":"62a5e4d4945edfb93a33059d07264cfa11aa50b1cad181388a41bdf5dd41a48e"}
Oct 30 12:48:46.973434 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.973013    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd" event={"ID":"0aff0921-3696-4f79-8380-f4bd418fde32","Type":"ContainerStarted","Data":"73fd6f869aca6459a4a5ff464beb4e5118e15b66545426558f1a7fdff981596c"}
Oct 30 12:48:46.973434 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:46.973051    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:48:47.022901 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:47.022846    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd" podStartSLOduration=2.022832336 podStartE2EDuration="2.022832336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:48:47.020405416 +0000 UTC m=+991.496125026" watchObservedRunningTime="2026-10-30 12:48:47.022832336 +0000 UTC m=+991.498551924"
Oct 30 12:48:47.690701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:47.690662    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f5b18c-669d-45d7-a435-cc2028580edf" path="/var/lib/kubelet/pods/85f5b18c-669d-45d7-a435-cc2028580edf/volumes"
Oct 30 12:48:57.980200 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:48:57.980167    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7hpfd"
Oct 30 12:49:05.995970 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:05.995933    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"]
Oct 30 12:49:05.996404 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:05.996349    4993 cpu_manager.go:414] "containerMap: removing container" podUID="85f5b18c-669d-45d7-a435-cc2028580edf" containerName="manager"
Oct 30 12:49:05.996404 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:05.996361    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f5b18c-669d-45d7-a435-cc2028580edf" containerName="manager"
Oct 30 12:49:05.996494 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:05.996425    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f5b18c-669d-45d7-a435-cc2028580edf" containerName="manager"
Oct 30 12:49:06.003529 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.003499    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.019678 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.019642    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"]
Oct 30 12:49:06.030797 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.030760    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 12:49:06.034078 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.034049    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cmshn\""
Oct 30 12:49:06.043370 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.043329    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 12:49:06.120704 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.120663    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd04f389-4aae-4a44-882c-be7855d43b44-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.120704 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.120707    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd04f389-4aae-4a44-882c-be7855d43b44-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.120970 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.120859    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rz5\" (UniqueName: \"kubernetes.io/projected/cd04f389-4aae-4a44-882c-be7855d43b44-kube-api-access-f2rz5\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.221389 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.221346    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rz5\" (UniqueName: \"kubernetes.io/projected/cd04f389-4aae-4a44-882c-be7855d43b44-kube-api-access-f2rz5\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.221607 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.221445    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd04f389-4aae-4a44-882c-be7855d43b44-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.221607 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.221469    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd04f389-4aae-4a44-882c-be7855d43b44-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.222167 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.222139    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd04f389-4aae-4a44-882c-be7855d43b44-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.224066 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.224044    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd04f389-4aae-4a44-882c-be7855d43b44-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.246639 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.246565    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rz5\" (UniqueName: \"kubernetes.io/projected/cd04f389-4aae-4a44-882c-be7855d43b44-kube-api-access-f2rz5\") pod \"kuadrant-console-plugin-66c4fb575d-dn49h\" (UID: \"cd04f389-4aae-4a44-882c-be7855d43b44\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.327918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.327860    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"
Oct 30 12:49:06.634919 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.634893    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h"]
Oct 30 12:49:06.637542 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:49:06.637511    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd04f389_4aae_4a44_882c_be7855d43b44.slice/crio-56116e3bb1df66de5bdd70e46fdf369d7f261fef07d22f158a9cde4a2601762c WatchSource:0}: Error finding container 56116e3bb1df66de5bdd70e46fdf369d7f261fef07d22f158a9cde4a2601762c: Status 404 returned error can't find the container with id 56116e3bb1df66de5bdd70e46fdf369d7f261fef07d22f158a9cde4a2601762c
Oct 30 12:49:06.639405 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:06.639383    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:49:07.081821 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.081774    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h" event={"ID":"cd04f389-4aae-4a44-882c-be7855d43b44","Type":"ContainerStarted","Data":"56116e3bb1df66de5bdd70e46fdf369d7f261fef07d22f158a9cde4a2601762c"}
Oct 30 12:49:07.340050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.339540    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v59pb"]
Oct 30 12:49:07.343913 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.343883    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb"
Oct 30 12:49:07.361377 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.360888    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vgvgv\""
Oct 30 12:49:07.377666 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.377625    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v59pb"]
Oct 30 12:49:07.430784 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.430747    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvfd\" (UniqueName: \"kubernetes.io/projected/50cc4464-4876-4c61-acaf-acd6f4eebb18-kube-api-access-sqvfd\") pod \"authorino-56c6c9ddd5-v59pb\" (UID: \"50cc4464-4876-4c61-acaf-acd6f4eebb18\") " pod="kuadrant-system/authorino-56c6c9ddd5-v59pb"
Oct 30 12:49:07.532176 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.532140    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvfd\" (UniqueName: \"kubernetes.io/projected/50cc4464-4876-4c61-acaf-acd6f4eebb18-kube-api-access-sqvfd\") pod \"authorino-56c6c9ddd5-v59pb\" (UID: \"50cc4464-4876-4c61-acaf-acd6f4eebb18\") " pod="kuadrant-system/authorino-56c6c9ddd5-v59pb"
Oct 30 12:49:07.558297 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.558258    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvfd\" (UniqueName: \"kubernetes.io/projected/50cc4464-4876-4c61-acaf-acd6f4eebb18-kube-api-access-sqvfd\") pod \"authorino-56c6c9ddd5-v59pb\" (UID: \"50cc4464-4876-4c61-acaf-acd6f4eebb18\") " pod="kuadrant-system/authorino-56c6c9ddd5-v59pb"
Oct 30 12:49:07.664254 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.664135    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb"
Oct 30 12:49:07.992981 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:07.992945    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v59pb"]
Oct 30 12:49:07.994482 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:49:07.994448    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50cc4464_4876_4c61_acaf_acd6f4eebb18.slice/crio-07920b7b2ea04c99edaa7207627e717151f0cd2b048c4dbb5edf4fb659283d71 WatchSource:0}: Error finding container 07920b7b2ea04c99edaa7207627e717151f0cd2b048c4dbb5edf4fb659283d71: Status 404 returned error can't find the container with id 07920b7b2ea04c99edaa7207627e717151f0cd2b048c4dbb5edf4fb659283d71
Oct 30 12:49:08.091942 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:08.091901    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb" event={"ID":"50cc4464-4876-4c61-acaf-acd6f4eebb18","Type":"ContainerStarted","Data":"07920b7b2ea04c99edaa7207627e717151f0cd2b048c4dbb5edf4fb659283d71"}
Oct 30 12:49:08.577893 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:49:08.577836    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:49:10.111874 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:10.111831    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb" event={"ID":"50cc4464-4876-4c61-acaf-acd6f4eebb18","Type":"ContainerStarted","Data":"7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c"}
Oct 30 12:49:10.155366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:10.155307    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb" podStartSLOduration=1.527123772 podStartE2EDuration="3.155293058s" totalImagesPullingTime="1.628169286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:10.154423357 +0000 UTC m=+1014.630142969" watchObservedRunningTime="2026-10-30 12:49:10.155293058 +0000 UTC m=+1014.631012669"
Oct 30 12:49:12.124851 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:12.124805    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h" event={"ID":"cd04f389-4aae-4a44-882c-be7855d43b44","Type":"ContainerStarted","Data":"b15dcd8c71bea3fbbaf05cd01ed5d47269371ec4f7ac994066c23e7dddc55f28"}
Oct 30 12:49:12.168342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:12.168282    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dn49h" podStartSLOduration=2.630257945 podStartE2EDuration="7.168265923s" totalImagesPullingTime="4.538007978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:12.167523311 +0000 UTC m=+1016.643242921" watchObservedRunningTime="2026-10-30 12:49:12.168265923 +0000 UTC m=+1016.643985533"
Oct 30 12:49:38.569794 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:49:38.569754    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:49:45.353928 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.353889    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85cd47f586-hfkxp"]
Oct 30 12:49:45.370956 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.370919    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.381191 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.381161    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85cd47f586-hfkxp"]
Oct 30 12:49:45.392159 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.392122    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 12:49:45.401366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.401332    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 12:49:45.401366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.401341    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-75gjs\""
Oct 30 12:49:45.401568 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.401392    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 12:49:45.481731 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.481687    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgq9\" (UniqueName: \"kubernetes.io/projected/360941cf-72a2-47d2-b73d-c4231f4a3e7c-kube-api-access-ckgq9\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.481731 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.481738    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-webhook-cert\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.481950 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.481887    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-metrics-certs\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.583153 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.583115    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-metrics-certs\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.583348 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.583201    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgq9\" (UniqueName: \"kubernetes.io/projected/360941cf-72a2-47d2-b73d-c4231f4a3e7c-kube-api-access-ckgq9\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.583348 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.583284    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-webhook-cert\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.585808 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.585780    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-webhook-cert\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.585927 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.585899    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-metrics-certs\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.608966 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.608880    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgq9\" (UniqueName: \"kubernetes.io/projected/360941cf-72a2-47d2-b73d-c4231f4a3e7c-kube-api-access-ckgq9\") pod \"maas-controller-85cd47f586-hfkxp\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") " pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.686281 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.686247    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:45.995430 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:45.995399    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85cd47f586-hfkxp"]
Oct 30 12:49:45.997720 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:49:45.997690    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360941cf_72a2_47d2_b73d_c4231f4a3e7c.slice/crio-6226a46b9194350a345be91cee6db12549fd3978dfe423bad951a94d613fd820 WatchSource:0}: Error finding container 6226a46b9194350a345be91cee6db12549fd3978dfe423bad951a94d613fd820: Status 404 returned error can't find the container with id 6226a46b9194350a345be91cee6db12549fd3978dfe423bad951a94d613fd820
Oct 30 12:49:46.303514 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:46.303421    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85cd47f586-hfkxp" event={"ID":"360941cf-72a2-47d2-b73d-c4231f4a3e7c","Type":"ContainerStarted","Data":"6226a46b9194350a345be91cee6db12549fd3978dfe423bad951a94d613fd820"}
Oct 30 12:49:49.324429 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:49.324384    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85cd47f586-hfkxp" event={"ID":"360941cf-72a2-47d2-b73d-c4231f4a3e7c","Type":"ContainerStarted","Data":"6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b"}
Oct 30 12:49:49.324840 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:49.324557    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:49:49.412831 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:49:49.412773    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-85cd47f586-hfkxp" podStartSLOduration=1.954791832 podStartE2EDuration="4.412757624s" totalImagesPullingTime="2.457965792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:49.412679746 +0000 UTC m=+1053.888399357" watchObservedRunningTime="2026-10-30 12:49:49.412757624 +0000 UTC m=+1053.888477234"
Oct 30 12:50:00.289814 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.289761    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85cd47f586-hfkxp"]
Oct 30 12:50:00.290293 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.290019    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-85cd47f586-hfkxp" podUID="360941cf-72a2-47d2-b73d-c4231f4a3e7c" containerName="manager" containerID="cri-o://6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b" gracePeriod=30
Oct 30 12:50:00.297343 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.297302    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:50:00.553061 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.552987    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:50:00.730023 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.729980    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-webhook-cert\") pod \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") "
Oct 30 12:50:00.730183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.730072    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360941cf-72a2-47d2-b73d-c4231f4a3e7c-kube-api-access-ckgq9\" (UniqueName: \"kubernetes.io/projected/360941cf-72a2-47d2-b73d-c4231f4a3e7c-kube-api-access-ckgq9\") pod \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") "
Oct 30 12:50:00.730183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.730170    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-metrics-certs\") pod \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\" (UID: \"360941cf-72a2-47d2-b73d-c4231f4a3e7c\") "
Oct 30 12:50:00.732508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.732467    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-webhook-cert" pod "360941cf-72a2-47d2-b73d-c4231f4a3e7c" (UID: "360941cf-72a2-47d2-b73d-c4231f4a3e7c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:50:00.732508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.732490    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-metrics-certs" pod "360941cf-72a2-47d2-b73d-c4231f4a3e7c" (UID: "360941cf-72a2-47d2-b73d-c4231f4a3e7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:50:00.732685 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.732512    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360941cf-72a2-47d2-b73d-c4231f4a3e7c-kube-api-access-ckgq9" pod "360941cf-72a2-47d2-b73d-c4231f4a3e7c" (UID: "360941cf-72a2-47d2-b73d-c4231f4a3e7c"). InnerVolumeSpecName "kube-api-access-ckgq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:50:00.831257 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.831131    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-metrics-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:50:00.831257 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.831165    4993 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/360941cf-72a2-47d2-b73d-c4231f4a3e7c-webhook-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:50:00.831257 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:00.831174    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckgq9\" (UniqueName: \"kubernetes.io/projected/360941cf-72a2-47d2-b73d-c4231f4a3e7c-kube-api-access-ckgq9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:50:01.391019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.390983    4993 generic.go:356] "Generic (PLEG): container finished" podID="360941cf-72a2-47d2-b73d-c4231f4a3e7c" containerID="6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b" exitCode=0
Oct 30 12:50:01.391489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.391047    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85cd47f586-hfkxp"
Oct 30 12:50:01.391489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.391075    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85cd47f586-hfkxp" event={"ID":"360941cf-72a2-47d2-b73d-c4231f4a3e7c","Type":"ContainerDied","Data":"6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b"}
Oct 30 12:50:01.391489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.391131    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85cd47f586-hfkxp" event={"ID":"360941cf-72a2-47d2-b73d-c4231f4a3e7c","Type":"ContainerDied","Data":"6226a46b9194350a345be91cee6db12549fd3978dfe423bad951a94d613fd820"}
Oct 30 12:50:01.391489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.391153    4993 scope.go:122] "RemoveContainer" containerID="6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b"
Oct 30 12:50:01.401125 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.400924    4993 scope.go:122] "RemoveContainer" containerID="6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b"
Oct 30 12:50:01.401379 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:50:01.401350    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b\": container with ID starting with 6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b not found: ID does not exist" containerID="6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b"
Oct 30 12:50:01.401492 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.401389    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b"} err="failed to get container status \"6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b\": rpc error: code = NotFound desc = could not find container \"6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b\": container with ID starting with 6e6c32b2f088c21ddd5309667fa5165ff54e64613c1914f0196e74417b739c3b not found: ID does not exist"
Oct 30 12:50:01.448462 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.448425    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85cd47f586-hfkxp"]
Oct 30 12:50:01.462356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.462317    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85cd47f586-hfkxp"]
Oct 30 12:50:01.690322 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:01.690227    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360941cf-72a2-47d2-b73d-c4231f4a3e7c" path="/var/lib/kubelet/pods/360941cf-72a2-47d2-b73d-c4231f4a3e7c/volumes"
Oct 30 12:50:08.567031 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:50:08.566991    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:50:30.069054 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.069005    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-696qq"]
Oct 30 12:50:30.069682 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.069658    4993 cpu_manager.go:414] "containerMap: removing container" podUID="360941cf-72a2-47d2-b73d-c4231f4a3e7c" containerName="manager"
Oct 30 12:50:30.069746 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.069687    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360941cf-72a2-47d2-b73d-c4231f4a3e7c" containerName="manager"
Oct 30 12:50:30.069835 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.069821    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="360941cf-72a2-47d2-b73d-c4231f4a3e7c" containerName="manager"
Oct 30 12:50:30.076892 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.076859    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-696qq"
Oct 30 12:50:30.091032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.091000    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-696qq"]
Oct 30 12:50:30.216656 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.216619    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zpn\" (UniqueName: \"kubernetes.io/projected/bf87d112-d6f8-4194-9889-868ec4ca12d3-kube-api-access-58zpn\") pod \"authorino-6f6944bcdd-696qq\" (UID: \"bf87d112-d6f8-4194-9889-868ec4ca12d3\") " pod="kuadrant-system/authorino-6f6944bcdd-696qq"
Oct 30 12:50:30.236447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.236403    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:50:30.236617 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.236494    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:50:30.241039 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.241003    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-696qq"]
Oct 30 12:50:30.241315 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:50:30.241290    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-58zpn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-696qq" podUID="bf87d112-d6f8-4194-9889-868ec4ca12d3"
Oct 30 12:50:30.317854 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.317808    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zpn\" (UniqueName: \"kubernetes.io/projected/bf87d112-d6f8-4194-9889-868ec4ca12d3-kube-api-access-58zpn\") pod \"authorino-6f6944bcdd-696qq\" (UID: \"bf87d112-d6f8-4194-9889-868ec4ca12d3\") " pod="kuadrant-system/authorino-6f6944bcdd-696qq"
Oct 30 12:50:30.345448 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.345366    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zpn\" (UniqueName: \"kubernetes.io/projected/bf87d112-d6f8-4194-9889-868ec4ca12d3-kube-api-access-58zpn\") pod \"authorino-6f6944bcdd-696qq\" (UID: \"bf87d112-d6f8-4194-9889-868ec4ca12d3\") " pod="kuadrant-system/authorino-6f6944bcdd-696qq"
Oct 30 12:50:30.433315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.433274    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bd48f4-r2nh8"]
Oct 30 12:50:30.436973 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.436950    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:50:30.456566 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.456512    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 12:50:30.458833 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.458800    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bd48f4-r2nh8"]
Oct 30 12:50:30.519140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.519104    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbzh\" (UniqueName: \"kubernetes.io/projected/e570822d-27a0-4080-bc1e-c2f99140ce0a-kube-api-access-4mbzh\") pod \"authorino-6bd48f4-r2nh8\" (UID: \"e570822d-27a0-4080-bc1e-c2f99140ce0a\") " pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:50:30.519140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.519156    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e570822d-27a0-4080-bc1e-c2f99140ce0a-tls-cert\") pod \"authorino-6bd48f4-r2nh8\" (UID: \"e570822d-27a0-4080-bc1e-c2f99140ce0a\") " pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:50:30.546216 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.546173    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-696qq"
Oct 30 12:50:30.555105 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.555068    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-696qq"
Oct 30 12:50:30.620718 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.620622    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf87d112-d6f8-4194-9889-868ec4ca12d3-kube-api-access-58zpn\" (UniqueName: \"kubernetes.io/projected/bf87d112-d6f8-4194-9889-868ec4ca12d3-kube-api-access-58zpn\") pod \"bf87d112-d6f8-4194-9889-868ec4ca12d3\" (UID: \"bf87d112-d6f8-4194-9889-868ec4ca12d3\") "
Oct 30 12:50:30.620902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.620823    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbzh\" (UniqueName: \"kubernetes.io/projected/e570822d-27a0-4080-bc1e-c2f99140ce0a-kube-api-access-4mbzh\") pod \"authorino-6bd48f4-r2nh8\" (UID: \"e570822d-27a0-4080-bc1e-c2f99140ce0a\") " pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:50:30.620902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.620867    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e570822d-27a0-4080-bc1e-c2f99140ce0a-tls-cert\") pod \"authorino-6bd48f4-r2nh8\" (UID: \"e570822d-27a0-4080-bc1e-c2f99140ce0a\") " pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:50:30.622979 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.622945    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf87d112-d6f8-4194-9889-868ec4ca12d3-kube-api-access-58zpn" pod "bf87d112-d6f8-4194-9889-868ec4ca12d3" (UID: "bf87d112-d6f8-4194-9889-868ec4ca12d3"). InnerVolumeSpecName "kube-api-access-58zpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:50:30.623466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.623446    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e570822d-27a0-4080-bc1e-c2f99140ce0a-tls-cert\") pod \"authorino-6bd48f4-r2nh8\" (UID: \"e570822d-27a0-4080-bc1e-c2f99140ce0a\") " pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:50:30.658071 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.658042    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbzh\" (UniqueName: \"kubernetes.io/projected/e570822d-27a0-4080-bc1e-c2f99140ce0a-kube-api-access-4mbzh\") pod \"authorino-6bd48f4-r2nh8\" (UID: \"e570822d-27a0-4080-bc1e-c2f99140ce0a\") " pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:50:30.721939 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.721893    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zpn\" (UniqueName: \"kubernetes.io/projected/bf87d112-d6f8-4194-9889-868ec4ca12d3-kube-api-access-58zpn\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:50:30.750974 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:30.750934    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:50:31.051860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.051829    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bd48f4-r2nh8"]
Oct 30 12:50:31.054678 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:50:31.054646    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode570822d_27a0_4080_bc1e_c2f99140ce0a.slice/crio-4b6d155313f438d495e2615fd11a0fb65ec21d03a677ee8c73dd8eae281f578a WatchSource:0}: Error finding container 4b6d155313f438d495e2615fd11a0fb65ec21d03a677ee8c73dd8eae281f578a: Status 404 returned error can't find the container with id 4b6d155313f438d495e2615fd11a0fb65ec21d03a677ee8c73dd8eae281f578a
Oct 30 12:50:31.552392 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.552353    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bd48f4-r2nh8" event={"ID":"e570822d-27a0-4080-bc1e-c2f99140ce0a","Type":"ContainerStarted","Data":"4b6d155313f438d495e2615fd11a0fb65ec21d03a677ee8c73dd8eae281f578a"}
Oct 30 12:50:31.553950 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.553926    4993 generic.go:356] "Generic (PLEG): container finished" podID="d33694bd-a804-4182-8d4b-e30156066d8f" containerID="d21bc8d958ecbc5231dc2152a9a388174a8a85f223f853be8f053aaf2878fc0c" exitCode=0
Oct 30 12:50:31.554052 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.554002    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" event={"ID":"d33694bd-a804-4182-8d4b-e30156066d8f","Type":"ContainerDied","Data":"d21bc8d958ecbc5231dc2152a9a388174a8a85f223f853be8f053aaf2878fc0c"}
Oct 30 12:50:31.554052 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.554020    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-696qq"
Oct 30 12:50:31.554422 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.554405    4993 scope.go:122] "RemoveContainer" containerID="d21bc8d958ecbc5231dc2152a9a388174a8a85f223f853be8f053aaf2878fc0c"
Oct 30 12:50:31.555477 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.555446    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:50:31.555539 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.555519    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:50:31.555593 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.555581    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:50:31.703455 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.703350    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-696qq"]
Oct 30 12:50:31.710082 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:31.709998    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-696qq"]
Oct 30 12:50:32.581709 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:32.581675    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bd48f4-r2nh8" event={"ID":"e570822d-27a0-4080-bc1e-c2f99140ce0a","Type":"ContainerStarted","Data":"c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9"}
Oct 30 12:50:32.583836 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:32.583810    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" event={"ID":"d33694bd-a804-4182-8d4b-e30156066d8f","Type":"ContainerStarted","Data":"b036d85758687f791a3d82e2312257e881843792167c2412449f0a927c1f1a59"}
Oct 30 12:50:32.626500 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:32.626435    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6bd48f4-r2nh8" podStartSLOduration=2.224975408 podStartE2EDuration="2.626414773s" totalImagesPullingTime="401.439365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:32.624940354 +0000 UTC m=+1097.100659966" watchObservedRunningTime="2026-10-30 12:50:32.626414773 +0000 UTC m=+1097.102134384"
Oct 30 12:50:32.688247 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:32.685577    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v59pb"]
Oct 30 12:50:32.688247 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:32.685822    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb" podUID="50cc4464-4876-4c61-acaf-acd6f4eebb18" containerName="authorino" containerID="cri-o://7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c" gracePeriod=30
Oct 30 12:50:32.986662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:32.986626    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb"
Oct 30 12:50:33.048884 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.048841    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cc4464-4876-4c61-acaf-acd6f4eebb18-kube-api-access-sqvfd\" (UniqueName: \"kubernetes.io/projected/50cc4464-4876-4c61-acaf-acd6f4eebb18-kube-api-access-sqvfd\") pod \"50cc4464-4876-4c61-acaf-acd6f4eebb18\" (UID: \"50cc4464-4876-4c61-acaf-acd6f4eebb18\") "
Oct 30 12:50:33.050885 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.050847    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cc4464-4876-4c61-acaf-acd6f4eebb18-kube-api-access-sqvfd" pod "50cc4464-4876-4c61-acaf-acd6f4eebb18" (UID: "50cc4464-4876-4c61-acaf-acd6f4eebb18"). InnerVolumeSpecName "kube-api-access-sqvfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:50:33.150515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.150431    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvfd\" (UniqueName: \"kubernetes.io/projected/50cc4464-4876-4c61-acaf-acd6f4eebb18-kube-api-access-sqvfd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:50:33.474193 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.473635    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:50:33.474193 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.473798    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:50:33.474193 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.473843    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 12:50:33.576525 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.576468    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"]
Oct 30 12:50:33.577396 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.577358    4993 cpu_manager.go:414] "containerMap: removing container" podUID="50cc4464-4876-4c61-acaf-acd6f4eebb18" containerName="authorino"
Oct 30 12:50:33.577396 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.577387    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cc4464-4876-4c61-acaf-acd6f4eebb18" containerName="authorino"
Oct 30 12:50:33.577598 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.577504    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cc4464-4876-4c61-acaf-acd6f4eebb18" containerName="authorino"
Oct 30 12:50:33.580643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.580617    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:33.590611 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.590582    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"]
Oct 30 12:50:33.591600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.591565    4993 generic.go:356] "Generic (PLEG): container finished" podID="50cc4464-4876-4c61-acaf-acd6f4eebb18" containerID="7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c" exitCode=0
Oct 30 12:50:33.591729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.591638    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb"
Oct 30 12:50:33.591729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.591647    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb" event={"ID":"50cc4464-4876-4c61-acaf-acd6f4eebb18","Type":"ContainerDied","Data":"7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c"}
Oct 30 12:50:33.591729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.591690    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v59pb" event={"ID":"50cc4464-4876-4c61-acaf-acd6f4eebb18","Type":"ContainerDied","Data":"07920b7b2ea04c99edaa7207627e717151f0cd2b048c4dbb5edf4fb659283d71"}
Oct 30 12:50:33.591729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.591714    4993 scope.go:122] "RemoveContainer" containerID="7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c"
Oct 30 12:50:33.605334 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.605309    4993 scope.go:122] "RemoveContainer" containerID="7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c"
Oct 30 12:50:33.605673 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:50:33.605650    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c\": container with ID starting with 7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c not found: ID does not exist" containerID="7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c"
Oct 30 12:50:33.605751 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.605682    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c"} err="failed to get container status \"7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c\": rpc error: code = NotFound desc = could not find container \"7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c\": container with ID starting with 7fd6d98d9acd35678293b4edd13f1a183cd7dcbb634d7c23ba302de83ea5148c not found: ID does not exist"
Oct 30 12:50:33.677821 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.677770    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdss\" (UniqueName: \"kubernetes.io/projected/0ae3b9f9-8971-4208-8518-b25746b98186-kube-api-access-5kdss\") pod \"llmisvc-controller-manager-6466c554d8-2ln8c\" (UID: \"0ae3b9f9-8971-4208-8518-b25746b98186\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:33.678007 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.677944    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae3b9f9-8971-4208-8518-b25746b98186-cert\") pod \"llmisvc-controller-manager-6466c554d8-2ln8c\" (UID: \"0ae3b9f9-8971-4208-8518-b25746b98186\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:33.678983 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.678960    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v59pb"]
Oct 30 12:50:33.681757 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.681733    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v59pb"]
Oct 30 12:50:33.690344 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.690314    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cc4464-4876-4c61-acaf-acd6f4eebb18" path="/var/lib/kubelet/pods/50cc4464-4876-4c61-acaf-acd6f4eebb18/volumes"
Oct 30 12:50:33.690653 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.690640    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf87d112-d6f8-4194-9889-868ec4ca12d3" path="/var/lib/kubelet/pods/bf87d112-d6f8-4194-9889-868ec4ca12d3/volumes"
Oct 30 12:50:33.778794 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.778743    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae3b9f9-8971-4208-8518-b25746b98186-cert\") pod \"llmisvc-controller-manager-6466c554d8-2ln8c\" (UID: \"0ae3b9f9-8971-4208-8518-b25746b98186\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:33.779011 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.778820    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdss\" (UniqueName: \"kubernetes.io/projected/0ae3b9f9-8971-4208-8518-b25746b98186-kube-api-access-5kdss\") pod \"llmisvc-controller-manager-6466c554d8-2ln8c\" (UID: \"0ae3b9f9-8971-4208-8518-b25746b98186\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:33.781321 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.781293    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae3b9f9-8971-4208-8518-b25746b98186-cert\") pod \"llmisvc-controller-manager-6466c554d8-2ln8c\" (UID: \"0ae3b9f9-8971-4208-8518-b25746b98186\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:33.804648 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.804615    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdss\" (UniqueName: \"kubernetes.io/projected/0ae3b9f9-8971-4208-8518-b25746b98186-kube-api-access-5kdss\") pod \"llmisvc-controller-manager-6466c554d8-2ln8c\" (UID: \"0ae3b9f9-8971-4208-8518-b25746b98186\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:33.899597 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:33.899556    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:34.223878 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:34.223831    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"]
Oct 30 12:50:34.226353 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:50:34.226317    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae3b9f9_8971_4208_8518_b25746b98186.slice/crio-e39cb3ab42753cfef1fa3a156eea396f581e1c909098f8fc91e4da1c34d661a8 WatchSource:0}: Error finding container e39cb3ab42753cfef1fa3a156eea396f581e1c909098f8fc91e4da1c34d661a8: Status 404 returned error can't find the container with id e39cb3ab42753cfef1fa3a156eea396f581e1c909098f8fc91e4da1c34d661a8
Oct 30 12:50:34.477314 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:34.472650    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:50:34.477314 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:34.472717    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 12:50:34.601968 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:34.601935    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c" event={"ID":"0ae3b9f9-8971-4208-8518-b25746b98186","Type":"ContainerStarted","Data":"e39cb3ab42753cfef1fa3a156eea396f581e1c909098f8fc91e4da1c34d661a8"}
Oct 30 12:50:35.474233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:35.469623    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:50:35.474233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:35.469687    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 12:50:35.609271 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:35.609236    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c" event={"ID":"0ae3b9f9-8971-4208-8518-b25746b98186","Type":"ContainerStarted","Data":"f77b7293aa6b28793558202c6a166489d54cff4c02824f1412181326223c070a"}
Oct 30 12:50:35.609680 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:35.609336    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:50:35.679516 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:35.679458    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c" podStartSLOduration=2.386061565 podStartE2EDuration="2.679442528s" totalImagesPullingTime="293.380963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:35.678541161 +0000 UTC m=+1100.154260771" watchObservedRunningTime="2026-10-30 12:50:35.679442528 +0000 UTC m=+1100.155162178"
Oct 30 12:50:36.467712 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:36.467670    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:50:36.468012 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:36.467977    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 12:50:37.468858 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:37.468813    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:50:37.469360 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:37.468887    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 12:50:38.468991 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:38.468960    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:50:38.469456 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:38.469263    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:50:38.469869 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:50:38.469851    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8"
Oct 30 12:50:38.567717 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:50:38.567681    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:51:06.617188 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:06.617157    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2ln8c"
Oct 30 12:51:06.698928 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:06.698890    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"]
Oct 30 12:51:06.699244 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:06.699194    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq" podUID="a6db5d16-3638-4b85-ae44-69fb57b3c77a" containerName="manager" containerID="cri-o://03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5" gracePeriod=30
Oct 30 12:51:06.952997 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:06.952968    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:51:07.081768 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.081725    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6db5d16-3638-4b85-ae44-69fb57b3c77a-cert\" (UniqueName: \"kubernetes.io/secret/a6db5d16-3638-4b85-ae44-69fb57b3c77a-cert\") pod \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\" (UID: \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\") "
Oct 30 12:51:07.081953 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.081830    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6db5d16-3638-4b85-ae44-69fb57b3c77a-kube-api-access-frpmw\" (UniqueName: \"kubernetes.io/projected/a6db5d16-3638-4b85-ae44-69fb57b3c77a-kube-api-access-frpmw\") pod \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\" (UID: \"a6db5d16-3638-4b85-ae44-69fb57b3c77a\") "
Oct 30 12:51:07.083937 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.083906    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6db5d16-3638-4b85-ae44-69fb57b3c77a-kube-api-access-frpmw" pod "a6db5d16-3638-4b85-ae44-69fb57b3c77a" (UID: "a6db5d16-3638-4b85-ae44-69fb57b3c77a"). InnerVolumeSpecName "kube-api-access-frpmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:51:07.084061 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.083906    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6db5d16-3638-4b85-ae44-69fb57b3c77a-cert" pod "a6db5d16-3638-4b85-ae44-69fb57b3c77a" (UID: "a6db5d16-3638-4b85-ae44-69fb57b3c77a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:51:07.182869 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.182787    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frpmw\" (UniqueName: \"kubernetes.io/projected/a6db5d16-3638-4b85-ae44-69fb57b3c77a-kube-api-access-frpmw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:51:07.182869 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.182815    4993 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db5d16-3638-4b85-ae44-69fb57b3c77a-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:51:07.780534 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.780492    4993 generic.go:356] "Generic (PLEG): container finished" podID="a6db5d16-3638-4b85-ae44-69fb57b3c77a" containerID="03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5" exitCode=0
Oct 30 12:51:07.780534 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.780546    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"
Oct 30 12:51:07.781073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.780582    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq" event={"ID":"a6db5d16-3638-4b85-ae44-69fb57b3c77a","Type":"ContainerDied","Data":"03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5"}
Oct 30 12:51:07.781073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.780639    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq" event={"ID":"a6db5d16-3638-4b85-ae44-69fb57b3c77a","Type":"ContainerDied","Data":"aed36845d7fbaa53c454ba44cf44bf8a4dc367c5bd9315ab10d4e193c4e59f6e"}
Oct 30 12:51:07.781073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.780661    4993 scope.go:122] "RemoveContainer" containerID="03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5"
Oct 30 12:51:07.790446 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.790423    4993 scope.go:122] "RemoveContainer" containerID="03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5"
Oct 30 12:51:07.790791 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:51:07.790765    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5\": container with ID starting with 03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5 not found: ID does not exist" containerID="03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5"
Oct 30 12:51:07.790897 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.790797    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5"} err="failed to get container status \"03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5\": rpc error: code = NotFound desc = could not find container \"03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5\": container with ID starting with 03f6496201c74dd8bd729c0db51d0c6a33e6950cebedd12452416c2f107479e5 not found: ID does not exist"
Oct 30 12:51:07.861521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.861481    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"]
Oct 30 12:51:07.870315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:07.870284    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8mvtq"]
Oct 30 12:51:08.568595 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:51:08.568553    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:51:09.690500 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:09.690458    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6db5d16-3638-4b85-ae44-69fb57b3c77a" path="/var/lib/kubelet/pods/a6db5d16-3638-4b85-ae44-69fb57b3c77a/volumes"
Oct 30 12:51:20.639633 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.639592    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"]
Oct 30 12:51:20.640030 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.639977    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a6db5d16-3638-4b85-ae44-69fb57b3c77a" containerName="manager"
Oct 30 12:51:20.640030 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.639991    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6db5d16-3638-4b85-ae44-69fb57b3c77a" containerName="manager"
Oct 30 12:51:20.640101 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.640083    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6db5d16-3638-4b85-ae44-69fb57b3c77a" containerName="manager"
Oct 30 12:51:20.644597 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.644573    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.675984 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.675943    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"]
Oct 30 12:51:20.812600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812555    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.812600 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812605    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.812830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812634    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgg6b\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-kube-api-access-qgg6b\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.812830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812724    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a788667-becf-429a-af19-7df65728c6bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.812830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812747    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a788667-becf-429a-af19-7df65728c6bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.812830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812767    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.812830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812784    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.812989 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812882    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.812989 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.812932    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.913651 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.913568    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.913651 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.913642    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.913866 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.913671    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.913866 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.913694    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgg6b\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-kube-api-access-qgg6b\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.913866 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.913760    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a788667-becf-429a-af19-7df65728c6bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.913866 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.913785    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a788667-becf-429a-af19-7df65728c6bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.913866 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.913816    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.914104 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.913840    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.914104 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.914040    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.914104 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.914069    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.914290 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.914163    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.914356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.914310    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.914413 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.914379    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.914602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.914580    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a788667-becf-429a-af19-7df65728c6bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.916258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.916238    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.916473 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.916454    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a788667-becf-429a-af19-7df65728c6bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.939120 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.939085    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.939304 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.939224    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgg6b\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-kube-api-access-qgg6b\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mzhxg\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:20.958032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:20.957990    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:21.279684 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:21.279651    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"]
Oct 30 12:51:21.281809 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:51:21.281778    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a788667_becf_429a_af19_7df65728c6bd.slice/crio-0c7561bed85e3e615d0ddd800cca82979a4acc5240de1389aaf9abba87cd2021 WatchSource:0}: Error finding container 0c7561bed85e3e615d0ddd800cca82979a4acc5240de1389aaf9abba87cd2021: Status 404 returned error can't find the container with id 0c7561bed85e3e615d0ddd800cca82979a4acc5240de1389aaf9abba87cd2021
Oct 30 12:51:21.283976 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:21.283945    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:51:21.284043 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:21.284012    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:51:21.284100 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:21.284040    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:51:21.881559 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:21.881518    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" event={"ID":"1a788667-becf-429a-af19-7df65728c6bd","Type":"ContainerStarted","Data":"60282eac81f101656848f91e10a17c22db2969fd53bd6f5385f7d098fd706ca7"}
Oct 30 12:51:21.882003 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:21.881580    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" event={"ID":"1a788667-becf-429a-af19-7df65728c6bd","Type":"ContainerStarted","Data":"0c7561bed85e3e615d0ddd800cca82979a4acc5240de1389aaf9abba87cd2021"}
Oct 30 12:51:21.975473 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:21.975439    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:22.978234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:22.978168    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mzhxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:51:22.978651 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:22.978272    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 12:51:23.972232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:23.971662    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mzhxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:51:23.972232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:23.971732    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 12:51:24.969682 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:24.969637    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mzhxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:51:24.970389 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:24.970354    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 12:51:25.973232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:25.965621    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mzhxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:51:25.973232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:25.965685    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 12:51:26.971235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:26.959655    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mzhxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:51:26.971235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:26.959723    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 12:51:27.962592 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:27.962551    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:27.963040 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:27.962854    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:27.963689 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:27.963673    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 12:51:28.027055 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:51:28.026988    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podStartSLOduration=8.026973516 podStartE2EDuration="8.026973516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:21.97153168 +0000 UTC m=+1146.447251293" watchObservedRunningTime="2026-10-30 12:51:28.026973516 +0000 UTC m=+1152.502693125"
Oct 30 12:51:38.567333 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:51:38.567292    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:52:08.628075 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:52:08.628000    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:52:15.817619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:15.817583    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:52:15.825640 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:15.825611    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:52:20.799432 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.799302    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"]
Oct 30 12:52:20.803164 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.803140    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.836907 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.836864    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"]
Oct 30 12:52:20.869416 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869383    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h942k\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-kube-api-access-h942k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.869602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869441    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.869602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869556    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.869602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869587    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.869755 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869604    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.869815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869740    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.869815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869798    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.869908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869831    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.869908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.869864    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971157 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971113    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971350 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971168    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971350 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971284    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971350 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971319    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971494 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971358    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971494 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971401    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971494 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971429    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971494 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971463    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971689 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971524    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h942k\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-kube-api-access-h942k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971689 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971621    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971698    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971779    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.971935 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.971912    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.972051 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.972030    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.973899 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.973877    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.974128 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.974110    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.996668 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.996635    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:20.996809 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:20.996727    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h942k\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-kube-api-access-h942k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qtjv7\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:21.118620 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:21.118535    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:21.438103 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:21.437979    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"]
Oct 30 12:52:21.440514 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:52:21.440485    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf5adae_c15d_4e5e_8c24_0edbebff5ef5.slice/crio-491bba2a8967de2c48c1709b15e8f4e5a90fa91f6522a407c247d9efe4684093 WatchSource:0}: Error finding container 491bba2a8967de2c48c1709b15e8f4e5a90fa91f6522a407c247d9efe4684093: Status 404 returned error can't find the container with id 491bba2a8967de2c48c1709b15e8f4e5a90fa91f6522a407c247d9efe4684093
Oct 30 12:52:21.442592 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:21.442560    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:52:21.442674 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:21.442630    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:52:21.442674 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:21.442660    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:52:22.180232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.178686    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6956fb6d85-krc2x"]
Oct 30 12:52:22.184062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.183596    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6956fb6d85-krc2x"
Oct 30 12:52:22.186672 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.186637    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6956fb6d85-krc2x"]
Oct 30 12:52:22.284314 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.284271    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2cxd\" (UniqueName: \"kubernetes.io/projected/862929ad-91d4-47d8-8740-21c32f987f11-kube-api-access-d2cxd\") pod \"authorino-6956fb6d85-krc2x\" (UID: \"862929ad-91d4-47d8-8740-21c32f987f11\") " pod="kuadrant-system/authorino-6956fb6d85-krc2x"
Oct 30 12:52:22.284506 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.284390    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/862929ad-91d4-47d8-8740-21c32f987f11-tls-cert\") pod \"authorino-6956fb6d85-krc2x\" (UID: \"862929ad-91d4-47d8-8740-21c32f987f11\") " pod="kuadrant-system/authorino-6956fb6d85-krc2x"
Oct 30 12:52:22.285017 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.284991    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" event={"ID":"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5","Type":"ContainerStarted","Data":"ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb"}
Oct 30 12:52:22.285175 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.285045    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" event={"ID":"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5","Type":"ContainerStarted","Data":"491bba2a8967de2c48c1709b15e8f4e5a90fa91f6522a407c247d9efe4684093"}
Oct 30 12:52:22.344411 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.344347    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podStartSLOduration=2.344327401 podStartE2EDuration="2.344327401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:22.343999862 +0000 UTC m=+1206.819719473" watchObservedRunningTime="2026-10-30 12:52:22.344327401 +0000 UTC m=+1206.820047012"
Oct 30 12:52:22.385144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.385100    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/862929ad-91d4-47d8-8740-21c32f987f11-tls-cert\") pod \"authorino-6956fb6d85-krc2x\" (UID: \"862929ad-91d4-47d8-8740-21c32f987f11\") " pod="kuadrant-system/authorino-6956fb6d85-krc2x"
Oct 30 12:52:22.385347 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.385287    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2cxd\" (UniqueName: \"kubernetes.io/projected/862929ad-91d4-47d8-8740-21c32f987f11-kube-api-access-d2cxd\") pod \"authorino-6956fb6d85-krc2x\" (UID: \"862929ad-91d4-47d8-8740-21c32f987f11\") " pod="kuadrant-system/authorino-6956fb6d85-krc2x"
Oct 30 12:52:22.387986 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.387952    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/862929ad-91d4-47d8-8740-21c32f987f11-tls-cert\") pod \"authorino-6956fb6d85-krc2x\" (UID: \"862929ad-91d4-47d8-8740-21c32f987f11\") " pod="kuadrant-system/authorino-6956fb6d85-krc2x"
Oct 30 12:52:22.412187 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.412135    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2cxd\" (UniqueName: \"kubernetes.io/projected/862929ad-91d4-47d8-8740-21c32f987f11-kube-api-access-d2cxd\") pod \"authorino-6956fb6d85-krc2x\" (UID: \"862929ad-91d4-47d8-8740-21c32f987f11\") " pod="kuadrant-system/authorino-6956fb6d85-krc2x"
Oct 30 12:52:22.501715 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.501671    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6956fb6d85-krc2x"
Oct 30 12:52:22.826971 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:22.826931    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6956fb6d85-krc2x"]
Oct 30 12:52:22.828558 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:52:22.828527    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod862929ad_91d4_47d8_8740_21c32f987f11.slice/crio-93606abaf581e5ae56f6b557c83437561cb3d951cce1ae0be5ae2a704be8f231 WatchSource:0}: Error finding container 93606abaf581e5ae56f6b557c83437561cb3d951cce1ae0be5ae2a704be8f231: Status 404 returned error can't find the container with id 93606abaf581e5ae56f6b557c83437561cb3d951cce1ae0be5ae2a704be8f231
Oct 30 12:52:23.119632 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:23.119529    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:23.121102 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:23.121069    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:52:23.121271 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:23.121132    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 12:52:23.292569 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:23.292530    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6956fb6d85-krc2x" event={"ID":"862929ad-91d4-47d8-8740-21c32f987f11","Type":"ContainerStarted","Data":"93606abaf581e5ae56f6b557c83437561cb3d951cce1ae0be5ae2a704be8f231"}
Oct 30 12:52:24.119547 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.119506    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:52:24.119728 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.119576    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 12:52:24.309484 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.309436    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6956fb6d85-krc2x" event={"ID":"862929ad-91d4-47d8-8740-21c32f987f11","Type":"ContainerStarted","Data":"f23f1bf93d3a2ede98e0604c6acdbc9e34c0f6a414b90d0eafdb48758edf2129"}
Oct 30 12:52:24.392445 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.392328    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6956fb6d85-krc2x" podStartSLOduration=1.778728755 podStartE2EDuration="2.39230566s" totalImagesPullingTime="613.576905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:24.369380899 +0000 UTC m=+1208.845100518" watchObservedRunningTime="2026-10-30 12:52:24.39230566 +0000 UTC m=+1208.868025271"
Oct 30 12:52:24.400108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.400060    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bd48f4-r2nh8"]
Oct 30 12:52:24.400979 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.400936    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6bd48f4-r2nh8" podUID="e570822d-27a0-4080-bc1e-c2f99140ce0a" containerName="authorino" containerID="cri-o://c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9" gracePeriod=30
Oct 30 12:52:24.794143 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.794102    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:52:24.917333 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.917232    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e570822d-27a0-4080-bc1e-c2f99140ce0a-kube-api-access-4mbzh\" (UniqueName: \"kubernetes.io/projected/e570822d-27a0-4080-bc1e-c2f99140ce0a-kube-api-access-4mbzh\") pod \"e570822d-27a0-4080-bc1e-c2f99140ce0a\" (UID: \"e570822d-27a0-4080-bc1e-c2f99140ce0a\") "
Oct 30 12:52:24.917333 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.917308    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e570822d-27a0-4080-bc1e-c2f99140ce0a-tls-cert\" (UniqueName: \"kubernetes.io/secret/e570822d-27a0-4080-bc1e-c2f99140ce0a-tls-cert\") pod \"e570822d-27a0-4080-bc1e-c2f99140ce0a\" (UID: \"e570822d-27a0-4080-bc1e-c2f99140ce0a\") "
Oct 30 12:52:24.920116 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.920078    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e570822d-27a0-4080-bc1e-c2f99140ce0a-kube-api-access-4mbzh" pod "e570822d-27a0-4080-bc1e-c2f99140ce0a" (UID: "e570822d-27a0-4080-bc1e-c2f99140ce0a"). InnerVolumeSpecName "kube-api-access-4mbzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:24.931826 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:24.931788    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e570822d-27a0-4080-bc1e-c2f99140ce0a-tls-cert" pod "e570822d-27a0-4080-bc1e-c2f99140ce0a" (UID: "e570822d-27a0-4080-bc1e-c2f99140ce0a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:25.018911 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.018841    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbzh\" (UniqueName: \"kubernetes.io/projected/e570822d-27a0-4080-bc1e-c2f99140ce0a-kube-api-access-4mbzh\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:52:25.018911 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.018882    4993 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e570822d-27a0-4080-bc1e-c2f99140ce0a-tls-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:52:25.119413 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.119355    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:52:25.119599 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.119438    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 12:52:25.316361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.316323    4993 generic.go:356] "Generic (PLEG): container finished" podID="e570822d-27a0-4080-bc1e-c2f99140ce0a" containerID="c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9" exitCode=0
Oct 30 12:52:25.316884 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.316401    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bd48f4-r2nh8"
Oct 30 12:52:25.316884 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.316413    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bd48f4-r2nh8" event={"ID":"e570822d-27a0-4080-bc1e-c2f99140ce0a","Type":"ContainerDied","Data":"c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9"}
Oct 30 12:52:25.316884 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.316461    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bd48f4-r2nh8" event={"ID":"e570822d-27a0-4080-bc1e-c2f99140ce0a","Type":"ContainerDied","Data":"4b6d155313f438d495e2615fd11a0fb65ec21d03a677ee8c73dd8eae281f578a"}
Oct 30 12:52:25.316884 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.316479    4993 scope.go:122] "RemoveContainer" containerID="c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9"
Oct 30 12:52:25.328378 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.328352    4993 scope.go:122] "RemoveContainer" containerID="c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9"
Oct 30 12:52:25.328736 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:52:25.328703    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9\": container with ID starting with c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9 not found: ID does not exist" containerID="c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9"
Oct 30 12:52:25.328863 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.328745    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9"} err="failed to get container status \"c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9\": rpc error: code = NotFound desc = could not find container \"c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9\": container with ID starting with c7e2691f665fa2f14e927ba1e4ed835541c9301e2964e300e62832ceb6be45a9 not found: ID does not exist"
Oct 30 12:52:25.424983 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.424943    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bd48f4-r2nh8"]
Oct 30 12:52:25.434277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.434240    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6bd48f4-r2nh8"]
Oct 30 12:52:25.696917 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:25.696835    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e570822d-27a0-4080-bc1e-c2f99140ce0a" path="/var/lib/kubelet/pods/e570822d-27a0-4080-bc1e-c2f99140ce0a/volumes"
Oct 30 12:52:26.119016 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:26.118924    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:52:26.119016 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:26.118991    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 12:52:27.119445 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:27.119411    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:52:27.120035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:27.120004    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 12:52:28.119771 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:28.119731    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:52:28.120309 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:28.119789    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 12:52:29.119631 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:29.119589    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:52:29.119824 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:29.119643    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 12:52:30.123775 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:30.123739    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:30.124172 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:30.124024    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:30.124701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:52:30.124681    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 12:52:38.567263 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:52:38.567200    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:53:03.463004 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.462965    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"]
Oct 30 12:53:03.463481 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.463413    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e570822d-27a0-4080-bc1e-c2f99140ce0a" containerName="authorino"
Oct 30 12:53:03.463481 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.463428    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e570822d-27a0-4080-bc1e-c2f99140ce0a" containerName="authorino"
Oct 30 12:53:03.463553 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.463516    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e570822d-27a0-4080-bc1e-c2f99140ce0a" containerName="authorino"
Oct 30 12:53:03.466627 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.466609    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:03.502361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.502331    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f1c4045f\""
Oct 30 12:53:03.502361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.502361    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f1c4045f-dockercfg-f4bp2\""
Oct 30 12:53:03.504840 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.504813    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"]
Oct 30 12:53:03.521823 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.521790    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f26b\" (UniqueName: \"kubernetes.io/projected/310517a1-0ded-4b4a-935a-a8e71cce1714-kube-api-access-7f26b\") pod \"payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj\" (UID: \"310517a1-0ded-4b4a-935a-a8e71cce1714\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:03.522068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.522035    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310517a1-0ded-4b4a-935a-a8e71cce1714-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj\" (UID: \"310517a1-0ded-4b4a-935a-a8e71cce1714\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:03.623370 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.623312    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f26b\" (UniqueName: \"kubernetes.io/projected/310517a1-0ded-4b4a-935a-a8e71cce1714-kube-api-access-7f26b\") pod \"payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj\" (UID: \"310517a1-0ded-4b4a-935a-a8e71cce1714\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:03.623578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.623390    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310517a1-0ded-4b4a-935a-a8e71cce1714-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj\" (UID: \"310517a1-0ded-4b4a-935a-a8e71cce1714\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:03.624127 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.624105    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310517a1-0ded-4b4a-935a-a8e71cce1714-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj\" (UID: \"310517a1-0ded-4b4a-935a-a8e71cce1714\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:03.649448 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.649418    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f26b\" (UniqueName: \"kubernetes.io/projected/310517a1-0ded-4b4a-935a-a8e71cce1714-kube-api-access-7f26b\") pod \"payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj\" (UID: \"310517a1-0ded-4b4a-935a-a8e71cce1714\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:03.779968 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:03.779932    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:04.113238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.113195    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"]
Oct 30 12:53:04.548495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.548453    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj" event={"ID":"310517a1-0ded-4b4a-935a-a8e71cce1714","Type":"ContainerStarted","Data":"961eaf9668decaaa6a7d28570d5f03b4cb146b543c3cbbbde7467181190712af"}
Oct 30 12:53:04.610332 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.608806    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"]
Oct 30 12:53:04.614952 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.614927    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.637363 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.637316    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-98f58556-openshift-default-dockercfg-nhsbj\""
Oct 30 12:53:04.653301 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.653265    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"]
Oct 30 12:53:04.736834 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.736793    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-certs\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.737031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.736858    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-data\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.737031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.736885    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istiod-ca-cert\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.737031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.736922    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-credential-socket\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.737031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.736963    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8qt4\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-kube-api-access-r8qt4\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.737031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.737000    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-envoy\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.737031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.737026    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-token\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.737393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.737086    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-podinfo\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.737393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.737132    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-socket\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838544    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qt4\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-kube-api-access-r8qt4\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838604    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-envoy\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838637    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-token\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838695    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-podinfo\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838740    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-socket\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838795    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-certs\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838834    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-data\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838861    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istiod-ca-cert\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.839063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.838899    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-credential-socket\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.840799 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.840036    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-credential-socket\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.840799 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.840555    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-socket\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.840799 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.840694    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-data\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.840799 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.840707    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-certs\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.841276 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.841251    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istiod-ca-cert\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.843291 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.843262    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-envoy\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.843409 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.843301    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-podinfo\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.867530 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.866969    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-token\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.868196 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.868167    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qt4\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-kube-api-access-r8qt4\") pod \"e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") " pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:04.935093 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:04.935056    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:05.288156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:05.288119    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"]
Oct 30 12:53:05.290079 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:05.290045    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee23bbfd_0a30_485b_a94e_d959d94a6c47.slice/crio-b31bdec95d58887926b0273522422d19917cf3aeebbb718ea2b8f9e3431919cd WatchSource:0}: Error finding container b31bdec95d58887926b0273522422d19917cf3aeebbb718ea2b8f9e3431919cd: Status 404 returned error can't find the container with id b31bdec95d58887926b0273522422d19917cf3aeebbb718ea2b8f9e3431919cd
Oct 30 12:53:05.294107 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:05.294066    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:05.295688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:05.294159    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:05.295688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:05.294201    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:05.560860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:05.560199    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" event={"ID":"ee23bbfd-0a30-485b-a94e-d959d94a6c47","Type":"ContainerStarted","Data":"8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223"}
Oct 30 12:53:05.560860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:05.560283    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" event={"ID":"ee23bbfd-0a30-485b-a94e-d959d94a6c47","Type":"ContainerStarted","Data":"b31bdec95d58887926b0273522422d19917cf3aeebbb718ea2b8f9e3431919cd"}
Oct 30 12:53:05.609816 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:05.609760    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podStartSLOduration=1.6097450279999999 podStartE2EDuration="1.609745028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:05.607775001 +0000 UTC m=+1250.083494612" watchObservedRunningTime="2026-10-30 12:53:05.609745028 +0000 UTC m=+1250.085464641"
Oct 30 12:53:05.935598 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:05.935518    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:06.178565 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.178530    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"]
Oct 30 12:53:06.191361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.191268    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:06.227253 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.224445    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"]
Oct 30 12:53:06.240447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.240419    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-873e4f54\""
Oct 30 12:53:06.241332 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.241315    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-873e4f54-dockercfg-cqgcg\""
Oct 30 12:53:06.259025 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.258983    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq2s7\" (UniqueName: \"kubernetes.io/projected/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-kube-api-access-dq2s7\") pod \"payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h\" (UID: \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:06.259187 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.259087    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h\" (UID: \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:06.360708 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.360667    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq2s7\" (UniqueName: \"kubernetes.io/projected/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-kube-api-access-dq2s7\") pod \"payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h\" (UID: \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:06.360900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.360773    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h\" (UID: \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:06.361877 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.361845    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h\" (UID: \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:06.387521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.387486    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq2s7\" (UniqueName: \"kubernetes.io/projected/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-kube-api-access-dq2s7\") pod \"payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h\" (UID: \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:06.541098 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.541059    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:06.594230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.591291    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj" event={"ID":"310517a1-0ded-4b4a-935a-a8e71cce1714","Type":"ContainerStarted","Data":"5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651"}
Oct 30 12:53:06.594230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.591397    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:06.646230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.646092    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj" podStartSLOduration=1.333719591 podStartE2EDuration="3.646072676s" totalImagesPullingTime="2.312353085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:06.644584884 +0000 UTC m=+1251.120304494" watchObservedRunningTime="2026-10-30 12:53:06.646072676 +0000 UTC m=+1251.121792287"
Oct 30 12:53:06.941817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.941783    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:06.956765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:06.956739    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"]
Oct 30 12:53:06.958878 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:06.958846    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c7d0c4_3ac0_4b91_975a_876c4e8ba8b5.slice/crio-1d5ba4e205ddf25482f5da5cdd5ef047cc503c17267b82d7407604674cae4abb WatchSource:0}: Error finding container 1d5ba4e205ddf25482f5da5cdd5ef047cc503c17267b82d7407604674cae4abb: Status 404 returned error can't find the container with id 1d5ba4e205ddf25482f5da5cdd5ef047cc503c17267b82d7407604674cae4abb
Oct 30 12:53:07.598926 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:07.598886    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h" event={"ID":"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5","Type":"ContainerStarted","Data":"e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2"}
Oct 30 12:53:07.599518 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:07.599001    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:07.599518 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:07.599030    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h" event={"ID":"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5","Type":"ContainerStarted","Data":"1d5ba4e205ddf25482f5da5cdd5ef047cc503c17267b82d7407604674cae4abb"}
Oct 30 12:53:07.599518 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:07.599145    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:07.600083 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:07.600034    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:07.642372 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:07.642324    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h" podStartSLOduration=1.6423109710000001 podStartE2EDuration="1.642310971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:07.641890075 +0000 UTC m=+1252.117609684" watchObservedRunningTime="2026-10-30 12:53:07.642310971 +0000 UTC m=+1252.118030582"
Oct 30 12:53:08.566925 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:53:08.566883    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:53:09.800980 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:09.797711    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"]
Oct 30 12:53:09.804050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:09.804022    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:09.821849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:09.821803    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"]
Oct 30 12:53:09.823340 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:09.823310    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-98f58556-dockercfg-vhg88\""
Oct 30 12:53:09.835875 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:09.835848    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-98f58556\""
Oct 30 12:53:09.926903 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:09.926864    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d3c282-e813-4727-a259-719025376289-plugins-config-volume\") pod \"payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr\" (UID: \"60d3c282-e813-4727-a259-719025376289\") " pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:09.926903 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:09.926911    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2xn\" (UniqueName: \"kubernetes.io/projected/60d3c282-e813-4727-a259-719025376289-kube-api-access-gf2xn\") pod \"payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr\" (UID: \"60d3c282-e813-4727-a259-719025376289\") " pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:10.027419 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.027369    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d3c282-e813-4727-a259-719025376289-plugins-config-volume\") pod \"payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr\" (UID: \"60d3c282-e813-4727-a259-719025376289\") " pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:10.027419 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.027418    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2xn\" (UniqueName: \"kubernetes.io/projected/60d3c282-e813-4727-a259-719025376289-kube-api-access-gf2xn\") pod \"payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr\" (UID: \"60d3c282-e813-4727-a259-719025376289\") " pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:10.028470 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.028441    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d3c282-e813-4727-a259-719025376289-plugins-config-volume\") pod \"payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr\" (UID: \"60d3c282-e813-4727-a259-719025376289\") " pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:10.053411 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.053335    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2xn\" (UniqueName: \"kubernetes.io/projected/60d3c282-e813-4727-a259-719025376289-kube-api-access-gf2xn\") pod \"payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr\" (UID: \"60d3c282-e813-4727-a259-719025376289\") " pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:10.138055 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.138018    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:10.446975 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.446949    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"]
Oct 30 12:53:10.449764 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:10.449733    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d3c282_e813_4727_a259_719025376289.slice/crio-4da29d41948f7ceffd00ccd4b8607bb0a49a11705fb0f70a30c6a1c4c2e7acc9 WatchSource:0}: Error finding container 4da29d41948f7ceffd00ccd4b8607bb0a49a11705fb0f70a30c6a1c4c2e7acc9: Status 404 returned error can't find the container with id 4da29d41948f7ceffd00ccd4b8607bb0a49a11705fb0f70a30c6a1c4c2e7acc9
Oct 30 12:53:10.619943 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.619892    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr" event={"ID":"60d3c282-e813-4727-a259-719025376289","Type":"ContainerStarted","Data":"d3e0843e6f22e1221370235e946fefb694c130a03ac475dbbfef967bc185e902"}
Oct 30 12:53:10.620118 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.619968    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr" event={"ID":"60d3c282-e813-4727-a259-719025376289","Type":"ContainerStarted","Data":"4da29d41948f7ceffd00ccd4b8607bb0a49a11705fb0f70a30c6a1c4c2e7acc9"}
Oct 30 12:53:10.620118 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.620002    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:10.670971 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:10.670912    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr" podStartSLOduration=1.670897512 podStartE2EDuration="1.670897512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:10.669594282 +0000 UTC m=+1255.145313895" watchObservedRunningTime="2026-10-30 12:53:10.670897512 +0000 UTC m=+1255.146617122"
Oct 30 12:53:18.609003 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:18.608974    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:18.609419 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:18.609128    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:19.322156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.322113    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"]
Oct 30 12:53:19.327809 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.327786    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.345447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.345413    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"]
Oct 30 12:53:19.364496 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.364436    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:53:19.365041 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.365023    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:53:19.365326 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.365311    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:53:19.367356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.367338    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:53:19.433255 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.433199    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7b59\" (UniqueName: \"kubernetes.io/projected/16aa58de-9d07-4525-8efe-d62034bbec1f-kube-api-access-k7b59\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.433569 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.433544    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.433720 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.433690    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.535222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.535181    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.535436 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.535249    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7b59\" (UniqueName: \"kubernetes.io/projected/16aa58de-9d07-4525-8efe-d62034bbec1f-kube-api-access-k7b59\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.535436 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.535322    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.536078 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.536047    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.538030 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.538004    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.575333 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.575252    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7b59\" (UniqueName: \"kubernetes.io/projected/16aa58de-9d07-4525-8efe-d62034bbec1f-kube-api-access-k7b59\") pod \"maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.642899 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.642864    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:19.953768 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:19.953713    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"]
Oct 30 12:53:19.955877 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:19.955848    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16aa58de_9d07_4525_8efe_d62034bbec1f.slice/crio-6970bb399ed3d682c848d09f906bad620483f32ce317ce5161c187964db47e3c WatchSource:0}: Error finding container 6970bb399ed3d682c848d09f906bad620483f32ce317ce5161c187964db47e3c: Status 404 returned error can't find the container with id 6970bb399ed3d682c848d09f906bad620483f32ce317ce5161c187964db47e3c
Oct 30 12:53:20.519470 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.519426    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"]
Oct 30 12:53:20.524674 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.524655    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.546122 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.546088    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzgv\" (UniqueName: \"kubernetes.io/projected/361b1696-fb06-4fb5-bbad-1645eec55948-kube-api-access-brzgv\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.546300 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.546193    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.546300 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.546244    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.547042 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.547017    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"]
Oct 30 12:53:20.647093 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.647055    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.647093 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.647094    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.647588 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.647266    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzgv\" (UniqueName: \"kubernetes.io/projected/361b1696-fb06-4fb5-bbad-1645eec55948-kube-api-access-brzgv\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.647867 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.647845    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.649513 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.649494    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.686661 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.686615    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzgv\" (UniqueName: \"kubernetes.io/projected/361b1696-fb06-4fb5-bbad-1645eec55948-kube-api-access-brzgv\") pod \"maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:20.701489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.701451    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw" event={"ID":"16aa58de-9d07-4525-8efe-d62034bbec1f","Type":"ContainerStarted","Data":"6970bb399ed3d682c848d09f906bad620483f32ce317ce5161c187964db47e3c"}
Oct 30 12:53:20.837810 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:20.837728    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:21.092975 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.090599    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"]
Oct 30 12:53:21.100004 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.099955    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"]
Oct 30 12:53:21.100229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.100119    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.149972 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.149938    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6j8\" (UniqueName: \"kubernetes.io/projected/d38924db-53c9-4088-821d-45ae7385ce8f-kube-api-access-nc6j8\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.150109 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.150038    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.150109 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.150091    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.186191 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.186156    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"]
Oct 30 12:53:21.187596 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:21.187562    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361b1696_fb06_4fb5_bbad_1645eec55948.slice/crio-4279d2d39409ece2bc62a7429779d532ba44f505842d8d416ec1698583d5d3f4 WatchSource:0}: Error finding container 4279d2d39409ece2bc62a7429779d532ba44f505842d8d416ec1698583d5d3f4: Status 404 returned error can't find the container with id 4279d2d39409ece2bc62a7429779d532ba44f505842d8d416ec1698583d5d3f4
Oct 30 12:53:21.251355 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.251313    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6j8\" (UniqueName: \"kubernetes.io/projected/d38924db-53c9-4088-821d-45ae7385ce8f-kube-api-access-nc6j8\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.251556 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.251430    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.251556 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.251486    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.260231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.252251    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.280599 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.280564    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.284663 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.284634    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6j8\" (UniqueName: \"kubernetes.io/projected/d38924db-53c9-4088-821d-45ae7385ce8f-kube-api-access-nc6j8\") pod \"maas-api-revoke-keys-e2e-ait-873e4f54-zms75\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.420734 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.420642    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:21.628660 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.628631    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:21.711264 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.711177    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx" event={"ID":"361b1696-fb06-4fb5-bbad-1645eec55948","Type":"ContainerStarted","Data":"4279d2d39409ece2bc62a7429779d532ba44f505842d8d416ec1698583d5d3f4"}
Oct 30 12:53:21.786103 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:21.786064    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"]
Oct 30 12:53:21.794901 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:21.794801    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38924db_53c9_4088_821d_45ae7385ce8f.slice/crio-db252eb379b10147e792056ab425f99aa8d0b01a369c8c01e1c0bc4538682bfe WatchSource:0}: Error finding container db252eb379b10147e792056ab425f99aa8d0b01a369c8c01e1c0bc4538682bfe: Status 404 returned error can't find the container with id db252eb379b10147e792056ab425f99aa8d0b01a369c8c01e1c0bc4538682bfe
Oct 30 12:53:22.720162 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:22.720114    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75" event={"ID":"d38924db-53c9-4088-821d-45ae7385ce8f","Type":"ContainerStarted","Data":"db252eb379b10147e792056ab425f99aa8d0b01a369c8c01e1c0bc4538682bfe"}
Oct 30 12:53:24.738227 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:24.738177    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw" event={"ID":"16aa58de-9d07-4525-8efe-d62034bbec1f","Type":"ContainerStarted","Data":"cc0de15f2572d7529c527b9f479462c26f881f0d73466ff370aa9e2e5ed98ce1"}
Oct 30 12:53:24.740266 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:24.740241    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75" event={"ID":"d38924db-53c9-4088-821d-45ae7385ce8f","Type":"ContainerStarted","Data":"d53b74b58f635bd682412a766bdca444391c920516c7147d9c5156b54f168973"}
Oct 30 12:53:24.802277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:24.802199    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw" podStartSLOduration=1.777560672 podStartE2EDuration="5.802183745s" totalImagesPullingTime="4.024623073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:24.801500792 +0000 UTC m=+1269.277220402" watchObservedRunningTime="2026-10-30 12:53:24.802183745 +0000 UTC m=+1269.277903386"
Oct 30 12:53:24.852008 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:24.850911    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75" podStartSLOduration=1.5066478700000001 podStartE2EDuration="3.850892318s" totalImagesPullingTime="2.344244448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:24.848481039 +0000 UTC m=+1269.324200648" watchObservedRunningTime="2026-10-30 12:53:24.850892318 +0000 UTC m=+1269.326611929"
Oct 30 12:53:25.747160 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:25.747067    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx" event={"ID":"361b1696-fb06-4fb5-bbad-1645eec55948","Type":"ContainerStarted","Data":"7a97d2ce5b073f16b9416d933935be4941a11bc553fc86faae385c492f2003f0"}
Oct 30 12:53:25.748795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:25.748765    4993 generic.go:356] "Generic (PLEG): container finished" podID="d38924db-53c9-4088-821d-45ae7385ce8f" containerID="d53b74b58f635bd682412a766bdca444391c920516c7147d9c5156b54f168973" exitCode=0
Oct 30 12:53:25.748910 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:25.748848    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75" event={"ID":"d38924db-53c9-4088-821d-45ae7385ce8f","Type":"ContainerDied","Data":"d53b74b58f635bd682412a766bdca444391c920516c7147d9c5156b54f168973"}
Oct 30 12:53:25.750418 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:25.750392    4993 generic.go:356] "Generic (PLEG): container finished" podID="16aa58de-9d07-4525-8efe-d62034bbec1f" containerID="cc0de15f2572d7529c527b9f479462c26f881f0d73466ff370aa9e2e5ed98ce1" exitCode=0
Oct 30 12:53:25.750502 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:25.750451    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw" event={"ID":"16aa58de-9d07-4525-8efe-d62034bbec1f","Type":"ContainerDied","Data":"cc0de15f2572d7529c527b9f479462c26f881f0d73466ff370aa9e2e5ed98ce1"}
Oct 30 12:53:25.791663 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:25.791604    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx" podStartSLOduration=1.7951852430000002 podStartE2EDuration="5.791584742s" totalImagesPullingTime="3.996399499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:25.790396823 +0000 UTC m=+1270.266116426" watchObservedRunningTime="2026-10-30 12:53:25.791584742 +0000 UTC m=+1270.267304353"
Oct 30 12:53:26.758784 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:26.758749    4993 generic.go:356] "Generic (PLEG): container finished" podID="361b1696-fb06-4fb5-bbad-1645eec55948" containerID="7a97d2ce5b073f16b9416d933935be4941a11bc553fc86faae385c492f2003f0" exitCode=0
Oct 30 12:53:26.759323 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:26.758834    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx" event={"ID":"361b1696-fb06-4fb5-bbad-1645eec55948","Type":"ContainerDied","Data":"7a97d2ce5b073f16b9416d933935be4941a11bc553fc86faae385c492f2003f0"}
Oct 30 12:53:26.969116 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:26.969084    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:26.972793 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:26.972767    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:27.056868 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.056777    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-openshift-ca\") pod \"d38924db-53c9-4088-821d-45ae7385ce8f\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") "
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057274    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-openshift-ca\") pod \"16aa58de-9d07-4525-8efe-d62034bbec1f\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") "
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057348    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16aa58de-9d07-4525-8efe-d62034bbec1f-kube-api-access-k7b59\" (UniqueName: \"kubernetes.io/projected/16aa58de-9d07-4525-8efe-d62034bbec1f-kube-api-access-k7b59\") pod \"16aa58de-9d07-4525-8efe-d62034bbec1f\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") "
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057383    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-xks-ca\") pod \"16aa58de-9d07-4525-8efe-d62034bbec1f\" (UID: \"16aa58de-9d07-4525-8efe-d62034bbec1f\") "
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057442    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38924db-53c9-4088-821d-45ae7385ce8f-kube-api-access-nc6j8\" (UniqueName: \"kubernetes.io/projected/d38924db-53c9-4088-821d-45ae7385ce8f-kube-api-access-nc6j8\") pod \"d38924db-53c9-4088-821d-45ae7385ce8f\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") "
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057540    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-openshift-ca" pod "d38924db-53c9-4088-821d-45ae7385ce8f" (UID: "d38924db-53c9-4088-821d-45ae7385ce8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057581    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-xks-ca\") pod \"d38924db-53c9-4088-821d-45ae7385ce8f\" (UID: \"d38924db-53c9-4088-821d-45ae7385ce8f\") "
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057587    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-openshift-ca" pod "16aa58de-9d07-4525-8efe-d62034bbec1f" (UID: "16aa58de-9d07-4525-8efe-d62034bbec1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057922    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:27.058964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.057941    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:27.061122 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.061092    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16aa58de-9d07-4525-8efe-d62034bbec1f-kube-api-access-k7b59" pod "16aa58de-9d07-4525-8efe-d62034bbec1f" (UID: "16aa58de-9d07-4525-8efe-d62034bbec1f"). InnerVolumeSpecName "kube-api-access-k7b59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:27.066077 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.061466    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38924db-53c9-4088-821d-45ae7385ce8f-kube-api-access-nc6j8" pod "d38924db-53c9-4088-821d-45ae7385ce8f" (UID: "d38924db-53c9-4088-821d-45ae7385ce8f"). InnerVolumeSpecName "kube-api-access-nc6j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:27.066077 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.061531    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-xks-ca" pod "16aa58de-9d07-4525-8efe-d62034bbec1f" (UID: "16aa58de-9d07-4525-8efe-d62034bbec1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:27.066077 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.065421    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-xks-ca" pod "d38924db-53c9-4088-821d-45ae7385ce8f" (UID: "d38924db-53c9-4088-821d-45ae7385ce8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:27.159433 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.159385    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38924db-53c9-4088-821d-45ae7385ce8f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:27.159433 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.159429    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7b59\" (UniqueName: \"kubernetes.io/projected/16aa58de-9d07-4525-8efe-d62034bbec1f-kube-api-access-k7b59\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:27.159433 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.159440    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16aa58de-9d07-4525-8efe-d62034bbec1f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:27.159664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.159450    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc6j8\" (UniqueName: \"kubernetes.io/projected/d38924db-53c9-4088-821d-45ae7385ce8f-kube-api-access-nc6j8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:27.766607 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.766576    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"
Oct 30 12:53:27.767029 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.766624    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw" event={"ID":"16aa58de-9d07-4525-8efe-d62034bbec1f","Type":"ContainerDied","Data":"6970bb399ed3d682c848d09f906bad620483f32ce317ce5161c187964db47e3c"}
Oct 30 12:53:27.767029 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.766694    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6970bb399ed3d682c848d09f906bad620483f32ce317ce5161c187964db47e3c"
Oct 30 12:53:27.768200 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.768177    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"
Oct 30 12:53:27.768355 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.768229    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75" event={"ID":"d38924db-53c9-4088-821d-45ae7385ce8f","Type":"ContainerDied","Data":"db252eb379b10147e792056ab425f99aa8d0b01a369c8c01e1c0bc4538682bfe"}
Oct 30 12:53:27.768355 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.768267    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db252eb379b10147e792056ab425f99aa8d0b01a369c8c01e1c0bc4538682bfe"
Oct 30 12:53:27.911719 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.911690    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:27.978426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.967683    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-xks-ca\") pod \"361b1696-fb06-4fb5-bbad-1645eec55948\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") "
Oct 30 12:53:27.978426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.968198    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-openshift-ca\") pod \"361b1696-fb06-4fb5-bbad-1645eec55948\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") "
Oct 30 12:53:27.978426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.968307    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361b1696-fb06-4fb5-bbad-1645eec55948-kube-api-access-brzgv\" (UniqueName: \"kubernetes.io/projected/361b1696-fb06-4fb5-bbad-1645eec55948-kube-api-access-brzgv\") pod \"361b1696-fb06-4fb5-bbad-1645eec55948\" (UID: \"361b1696-fb06-4fb5-bbad-1645eec55948\") "
Oct 30 12:53:27.978426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.969146    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-openshift-ca" pod "361b1696-fb06-4fb5-bbad-1645eec55948" (UID: "361b1696-fb06-4fb5-bbad-1645eec55948"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:27.978426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.971833    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-xks-ca" pod "361b1696-fb06-4fb5-bbad-1645eec55948" (UID: "361b1696-fb06-4fb5-bbad-1645eec55948"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:27.978426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:27.974876    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361b1696-fb06-4fb5-bbad-1645eec55948-kube-api-access-brzgv" pod "361b1696-fb06-4fb5-bbad-1645eec55948" (UID: "361b1696-fb06-4fb5-bbad-1645eec55948"). InnerVolumeSpecName "kube-api-access-brzgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:28.069805 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:28.069768    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:28.069805 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:28.069803    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361b1696-fb06-4fb5-bbad-1645eec55948-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:28.070006 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:28.069817    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brzgv\" (UniqueName: \"kubernetes.io/projected/361b1696-fb06-4fb5-bbad-1645eec55948-kube-api-access-brzgv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:28.777928 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:28.777892    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx" event={"ID":"361b1696-fb06-4fb5-bbad-1645eec55948","Type":"ContainerDied","Data":"4279d2d39409ece2bc62a7429779d532ba44f505842d8d416ec1698583d5d3f4"}
Oct 30 12:53:28.777928 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:28.777919    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"
Oct 30 12:53:28.778527 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:28.777941    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4279d2d39409ece2bc62a7429779d532ba44f505842d8d416ec1698583d5d3f4"
Oct 30 12:53:29.786487 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:29.786445    4993 generic.go:356] "Generic (PLEG): container finished" podID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerID="8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223" exitCode=0
Oct 30 12:53:29.786991 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:29.786526    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" event={"ID":"ee23bbfd-0a30-485b-a94e-d959d94a6c47","Type":"ContainerDied","Data":"8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223"}
Oct 30 12:53:29.787068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:29.786993    4993 scope.go:122] "RemoveContainer" containerID="8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223"
Oct 30 12:53:29.788502 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:29.788472    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:29.788620 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:29.788576    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:29.788691 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:29.788623    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:30.794496 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:30.794459    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" event={"ID":"ee23bbfd-0a30-485b-a94e-d959d94a6c47","Type":"ContainerStarted","Data":"0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1"}
Oct 30 12:53:30.935842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:30.935800    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:30.936060 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:30.936034    4993 patch_prober.go:29] interesting pod/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:53:30.936112 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:30.936085    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 12:53:31.016971 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:31.016937    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"]
Oct 30 12:53:31.850742 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:31.850706    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"]
Oct 30 12:53:31.936157 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:31.936121    4993 patch_prober.go:29] interesting pod/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:53:31.936353 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:31.936187    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 12:53:32.380407 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380372    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"]
Oct 30 12:53:32.380847 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380832    4993 cpu_manager.go:414] "containerMap: removing container" podUID="361b1696-fb06-4fb5-bbad-1645eec55948" containerName="revoke-keys"
Oct 30 12:53:32.380895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380849    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361b1696-fb06-4fb5-bbad-1645eec55948" containerName="revoke-keys"
Oct 30 12:53:32.380895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380865    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d38924db-53c9-4088-821d-45ae7385ce8f" containerName="revoke-keys"
Oct 30 12:53:32.380895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380870    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38924db-53c9-4088-821d-45ae7385ce8f" containerName="revoke-keys"
Oct 30 12:53:32.380895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380883    4993 cpu_manager.go:414] "containerMap: removing container" podUID="16aa58de-9d07-4525-8efe-d62034bbec1f" containerName="revoke-keys"
Oct 30 12:53:32.380895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380890    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16aa58de-9d07-4525-8efe-d62034bbec1f" containerName="revoke-keys"
Oct 30 12:53:32.381042 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380951    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38924db-53c9-4088-821d-45ae7385ce8f" containerName="revoke-keys"
Oct 30 12:53:32.381042 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380960    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="16aa58de-9d07-4525-8efe-d62034bbec1f" containerName="revoke-keys"
Oct 30 12:53:32.381042 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.380969    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="361b1696-fb06-4fb5-bbad-1645eec55948" containerName="revoke-keys"
Oct 30 12:53:32.386269 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.386251    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.406362 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.406333    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:53:32.406506 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.406373    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:53:32.406789 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.406769    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"]
Oct 30 12:53:32.413739 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.413706    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.413914 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.413890    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.413988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.413932    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjn54\" (UniqueName: \"kubernetes.io/projected/8d241014-9c85-4f86-98d7-9d08cc91826f-kube-api-access-wjn54\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.417782 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.417753    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:53:32.417880 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.417817    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:53:32.515366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.515324    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.515366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.515365    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjn54\" (UniqueName: \"kubernetes.io/projected/8d241014-9c85-4f86-98d7-9d08cc91826f-kube-api-access-wjn54\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.515595 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.515428    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.516082 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.516062    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.517788 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.517755    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.541637 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.541602    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjn54\" (UniqueName: \"kubernetes.io/projected/8d241014-9c85-4f86-98d7-9d08cc91826f-kube-api-access-wjn54\") pod \"maas-api-revoke-keys-e2e-derive-98f58556-92xw4\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.699448 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.699327    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:32.936523 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.936487    4993 patch_prober.go:29] interesting pod/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:53:32.937021 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:32.936562    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 12:53:33.035796 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.035758    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"]
Oct 30 12:53:33.041606 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:33.041576    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d241014_9c85_4f86_98d7_9d08cc91826f.slice/crio-bc835c9ddd86ee31408f9264a58cc3f75006b0710bfa1fcc09641812285c6951 WatchSource:0}: Error finding container bc835c9ddd86ee31408f9264a58cc3f75006b0710bfa1fcc09641812285c6951: Status 404 returned error can't find the container with id bc835c9ddd86ee31408f9264a58cc3f75006b0710bfa1fcc09641812285c6951
Oct 30 12:53:33.326565 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.326528    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"]
Oct 30 12:53:33.329243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.329188    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f1c4045f-hzkjx"]
Oct 30 12:53:33.690984 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.690895    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361b1696-fb06-4fb5-bbad-1645eec55948" path="/var/lib/kubelet/pods/361b1696-fb06-4fb5-bbad-1645eec55948/volumes"
Oct 30 12:53:33.803764 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.803620    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"]
Oct 30 12:53:33.806371 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.806337    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb8b5ba5-f7mgw"]
Oct 30 12:53:33.818505 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.818460    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4" event={"ID":"8d241014-9c85-4f86-98d7-9d08cc91826f","Type":"ContainerStarted","Data":"9bf8dcaf76073520ecf488d7f30c76a86bb27fb973234b9453557e89d420fa1d"}
Oct 30 12:53:33.818704 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.818534    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4" event={"ID":"8d241014-9c85-4f86-98d7-9d08cc91826f","Type":"ContainerStarted","Data":"bc835c9ddd86ee31408f9264a58cc3f75006b0710bfa1fcc09641812285c6951"}
Oct 30 12:53:33.873008 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.872942    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4" podStartSLOduration=1.872922125 podStartE2EDuration="1.872922125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:33.872637049 +0000 UTC m=+1278.348356671" watchObservedRunningTime="2026-10-30 12:53:33.872922125 +0000 UTC m=+1278.348641738"
Oct 30 12:53:33.935914 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.935868    4993 patch_prober.go:29] interesting pod/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:53:33.936079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:33.935947    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 12:53:34.827711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:34.827677    4993 generic.go:356] "Generic (PLEG): container finished" podID="8d241014-9c85-4f86-98d7-9d08cc91826f" containerID="9bf8dcaf76073520ecf488d7f30c76a86bb27fb973234b9453557e89d420fa1d" exitCode=0
Oct 30 12:53:34.828162 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:34.827764    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4" event={"ID":"8d241014-9c85-4f86-98d7-9d08cc91826f","Type":"ContainerDied","Data":"9bf8dcaf76073520ecf488d7f30c76a86bb27fb973234b9453557e89d420fa1d"}
Oct 30 12:53:34.935561 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:34.935526    4993 patch_prober.go:29] interesting pod/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:53:34.935750 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:34.935592    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 12:53:35.709564 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:35.709522    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16aa58de-9d07-4525-8efe-d62034bbec1f" path="/var/lib/kubelet/pods/16aa58de-9d07-4525-8efe-d62034bbec1f/volumes"
Oct 30 12:53:35.936929 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:35.936790    4993 patch_prober.go:29] interesting pod/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:53:35.936929 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:35.936860    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 12:53:35.984495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:35.984462    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"]
Oct 30 12:53:35.992196 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:35.992164    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-873e4f54-zms75"]
Oct 30 12:53:36.021246 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.017943    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj" podUID="310517a1-0ded-4b4a-935a-a8e71cce1714" containerName="payload-pre-processing" containerID="cri-o://5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651" gracePeriod=25
Oct 30 12:53:36.030130 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.030099    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:36.054310 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.054271    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-xks-ca\") pod \"8d241014-9c85-4f86-98d7-9d08cc91826f\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") "
Oct 30 12:53:36.054521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.054464    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d241014-9c85-4f86-98d7-9d08cc91826f-kube-api-access-wjn54\" (UniqueName: \"kubernetes.io/projected/8d241014-9c85-4f86-98d7-9d08cc91826f-kube-api-access-wjn54\") pod \"8d241014-9c85-4f86-98d7-9d08cc91826f\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") "
Oct 30 12:53:36.054521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.054502    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-openshift-ca\") pod \"8d241014-9c85-4f86-98d7-9d08cc91826f\" (UID: \"8d241014-9c85-4f86-98d7-9d08cc91826f\") "
Oct 30 12:53:36.055352 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.055305    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-openshift-ca" pod "8d241014-9c85-4f86-98d7-9d08cc91826f" (UID: "8d241014-9c85-4f86-98d7-9d08cc91826f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:36.056792 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.056767    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-xks-ca" pod "8d241014-9c85-4f86-98d7-9d08cc91826f" (UID: "8d241014-9c85-4f86-98d7-9d08cc91826f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:36.057851 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.057826    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d241014-9c85-4f86-98d7-9d08cc91826f-kube-api-access-wjn54" pod "8d241014-9c85-4f86-98d7-9d08cc91826f" (UID: "8d241014-9c85-4f86-98d7-9d08cc91826f"). InnerVolumeSpecName "kube-api-access-wjn54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:36.157269 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.157229    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjn54\" (UniqueName: \"kubernetes.io/projected/8d241014-9c85-4f86-98d7-9d08cc91826f-kube-api-access-wjn54\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:36.157269 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.157272    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:36.157470 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.157288    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d241014-9c85-4f86-98d7-9d08cc91826f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:36.318123 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.318087    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:36.358595 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.358556    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310517a1-0ded-4b4a-935a-a8e71cce1714-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310517a1-0ded-4b4a-935a-a8e71cce1714-plugins-config-volume\") pod \"310517a1-0ded-4b4a-935a-a8e71cce1714\" (UID: \"310517a1-0ded-4b4a-935a-a8e71cce1714\") "
Oct 30 12:53:36.358781 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.358681    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310517a1-0ded-4b4a-935a-a8e71cce1714-kube-api-access-7f26b\" (UniqueName: \"kubernetes.io/projected/310517a1-0ded-4b4a-935a-a8e71cce1714-kube-api-access-7f26b\") pod \"310517a1-0ded-4b4a-935a-a8e71cce1714\" (UID: \"310517a1-0ded-4b4a-935a-a8e71cce1714\") "
Oct 30 12:53:36.359055 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.359027    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310517a1-0ded-4b4a-935a-a8e71cce1714-plugins-config-volume" pod "310517a1-0ded-4b4a-935a-a8e71cce1714" (UID: "310517a1-0ded-4b4a-935a-a8e71cce1714"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:36.361164 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.361132    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310517a1-0ded-4b4a-935a-a8e71cce1714-kube-api-access-7f26b" pod "310517a1-0ded-4b4a-935a-a8e71cce1714" (UID: "310517a1-0ded-4b4a-935a-a8e71cce1714"). InnerVolumeSpecName "kube-api-access-7f26b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:36.460583 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.460540    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310517a1-0ded-4b4a-935a-a8e71cce1714-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:36.460583 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.460586    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f26b\" (UniqueName: \"kubernetes.io/projected/310517a1-0ded-4b4a-935a-a8e71cce1714-kube-api-access-7f26b\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:36.527537 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.527463    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"]
Oct 30 12:53:36.528240 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.528197    4993 cpu_manager.go:414] "containerMap: removing container" podUID="310517a1-0ded-4b4a-935a-a8e71cce1714" containerName="payload-pre-processing"
Oct 30 12:53:36.528240 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.528239    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310517a1-0ded-4b4a-935a-a8e71cce1714" containerName="payload-pre-processing"
Oct 30 12:53:36.528349 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.528253    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8d241014-9c85-4f86-98d7-9d08cc91826f" containerName="revoke-keys"
Oct 30 12:53:36.528349 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.528261    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d241014-9c85-4f86-98d7-9d08cc91826f" containerName="revoke-keys"
Oct 30 12:53:36.528406 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.528385    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="310517a1-0ded-4b4a-935a-a8e71cce1714" containerName="payload-pre-processing"
Oct 30 12:53:36.528406 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.528397    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d241014-9c85-4f86-98d7-9d08cc91826f" containerName="revoke-keys"
Oct 30 12:53:36.531023 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.531000    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.554004 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.553872    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e652ca-openshift-default-dockercfg-rr2pm\""
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.565951    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d5d695-584f-40a5-92a0-eb12f0b5b959-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.566015    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.566057    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.566113    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4kx\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-kube-api-access-vv4kx\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.566147    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.566191    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.566237    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.566306    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.568228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.566331    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.584962 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.581302    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"]
Oct 30 12:53:36.667100 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667062    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d5d695-584f-40a5-92a0-eb12f0b5b959-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667100 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667104    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667127    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667168    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4kx\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-kube-api-access-vv4kx\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667196    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667247    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667272    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667329    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667355    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667742 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667661    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667742 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667716    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.667960 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.667937    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.668149 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.668116    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d5d695-584f-40a5-92a0-eb12f0b5b959-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.668361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.668324    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.670036 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.670009    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.670318 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.670294    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.693241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.693190    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.693412 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.693391    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4kx\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-kube-api-access-vv4kx\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.843565 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.843471    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:36.847653 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.847612    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4" event={"ID":"8d241014-9c85-4f86-98d7-9d08cc91826f","Type":"ContainerDied","Data":"bc835c9ddd86ee31408f9264a58cc3f75006b0710bfa1fcc09641812285c6951"}
Oct 30 12:53:36.847788 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.847683    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc835c9ddd86ee31408f9264a58cc3f75006b0710bfa1fcc09641812285c6951"
Oct 30 12:53:36.847788 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.847769    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"
Oct 30 12:53:36.853835 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.853792    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h" podUID="50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5" containerName="payload-pre-processing" containerID="cri-o://e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2" gracePeriod=25
Oct 30 12:53:36.855602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.855574    4993 generic.go:356] "Generic (PLEG): container finished" podID="310517a1-0ded-4b4a-935a-a8e71cce1714" containerID="5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651" exitCode=0
Oct 30 12:53:36.855697 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.855684    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj" event={"ID":"310517a1-0ded-4b4a-935a-a8e71cce1714","Type":"ContainerDied","Data":"5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651"}
Oct 30 12:53:36.855766 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.855740    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj" event={"ID":"310517a1-0ded-4b4a-935a-a8e71cce1714","Type":"ContainerDied","Data":"961eaf9668decaaa6a7d28570d5f03b4cb146b543c3cbbbde7467181190712af"}
Oct 30 12:53:36.855823 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.855772    4993 scope.go:122] "RemoveContainer" containerID="5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651"
Oct 30 12:53:36.855949 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.855934    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"
Oct 30 12:53:36.925224 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.925182    4993 scope.go:122] "RemoveContainer" containerID="5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651"
Oct 30 12:53:36.925985 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:53:36.925904    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651\": container with ID starting with 5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651 not found: ID does not exist" containerID="5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651"
Oct 30 12:53:36.925985 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.925945    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651"} err="failed to get container status \"5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651\": rpc error: code = NotFound desc = could not find container \"5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651\": container with ID starting with 5336c9e616237320f2f2f5d95c2d0ca24b138cde86970328e845a614ec093651 not found: ID does not exist"
Oct 30 12:53:36.943916 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.943863    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:36.945403 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.945302    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:36.946362 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.946324    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:36.958730 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.958691    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"]
Oct 30 12:53:36.965363 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:36.965267    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f1c4045f-7fb96bbf5-c95gj"]
Oct 30 12:53:37.149189 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.149163    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:37.187964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.187937    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"]
Oct 30 12:53:37.190011 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:37.189985    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d5d695_584f_40a5_92a0_eb12f0b5b959.slice/crio-94027c6119737f8d14409010dbc8db9e9c49aab083d1d81d547f013a02a10f73 WatchSource:0}: Error finding container 94027c6119737f8d14409010dbc8db9e9c49aab083d1d81d547f013a02a10f73: Status 404 returned error can't find the container with id 94027c6119737f8d14409010dbc8db9e9c49aab083d1d81d547f013a02a10f73
Oct 30 12:53:37.192506 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.192469    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:37.192603 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.192536    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:37.192603 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.192573    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:37.274991 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.274963    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-kube-api-access-dq2s7\" (UniqueName: \"kubernetes.io/projected/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-kube-api-access-dq2s7\") pod \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\" (UID: \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\") "
Oct 30 12:53:37.275130 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.275084    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-plugins-config-volume\") pod \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\" (UID: \"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5\") "
Oct 30 12:53:37.275476 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.275454    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-plugins-config-volume" pod "50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5" (UID: "50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:37.277344 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.277319    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-kube-api-access-dq2s7" pod "50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5" (UID: "50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5"). InnerVolumeSpecName "kube-api-access-dq2s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:37.377032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.376941    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq2s7\" (UniqueName: \"kubernetes.io/projected/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-kube-api-access-dq2s7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:37.377032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.376979    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:37.693641 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.693556    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310517a1-0ded-4b4a-935a-a8e71cce1714" path="/var/lib/kubelet/pods/310517a1-0ded-4b4a-935a-a8e71cce1714/volumes"
Oct 30 12:53:37.694147 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.694129    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38924db-53c9-4088-821d-45ae7385ce8f" path="/var/lib/kubelet/pods/d38924db-53c9-4088-821d-45ae7385ce8f/volumes"
Oct 30 12:53:37.865720 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.865677    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" event={"ID":"02d5d695-584f-40a5-92a0-eb12f0b5b959","Type":"ContainerStarted","Data":"778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee"}
Oct 30 12:53:37.865914 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.865750    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" event={"ID":"02d5d695-584f-40a5-92a0-eb12f0b5b959","Type":"ContainerStarted","Data":"94027c6119737f8d14409010dbc8db9e9c49aab083d1d81d547f013a02a10f73"}
Oct 30 12:53:37.868439 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.868401    4993 generic.go:356] "Generic (PLEG): container finished" podID="50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5" containerID="e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2" exitCode=0
Oct 30 12:53:37.868619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.868447    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"
Oct 30 12:53:37.868619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.868548    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h" event={"ID":"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5","Type":"ContainerDied","Data":"e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2"}
Oct 30 12:53:37.868619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.868591    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h" event={"ID":"50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5","Type":"ContainerDied","Data":"1d5ba4e205ddf25482f5da5cdd5ef047cc503c17267b82d7407604674cae4abb"}
Oct 30 12:53:37.868780 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.868621    4993 scope.go:122] "RemoveContainer" containerID="e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2"
Oct 30 12:53:37.880695 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.880675    4993 scope.go:122] "RemoveContainer" containerID="e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2"
Oct 30 12:53:37.881166 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:53:37.881137    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2\": container with ID starting with e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2 not found: ID does not exist" containerID="e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2"
Oct 30 12:53:37.881271 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.881183    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2"} err="failed to get container status \"e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2\": rpc error: code = NotFound desc = could not find container \"e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2\": container with ID starting with e2ac7338907abf35bc42b48dcd39a72b8728bba6ec4d9d74cb261c1011b6b7b2 not found: ID does not exist"
Oct 30 12:53:37.921427 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.921378    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" podStartSLOduration=1.921361106 podStartE2EDuration="1.921361106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:37.918376255 +0000 UTC m=+1282.394095865" watchObservedRunningTime="2026-10-30 12:53:37.921361106 +0000 UTC m=+1282.397080716"
Oct 30 12:53:37.968269 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.968135    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"]
Oct 30 12:53:37.983281 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:37.983239    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-873e4f54-55948f5c59-wv89h"]
Oct 30 12:53:38.570703 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:53:38.570665    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:53:38.844900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:38.844816    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:38.850111 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:38.850081    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:38.878687 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:38.878652    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:38.879648 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:38.879629    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:53:39.694259 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:39.693993    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5" path="/var/lib/kubelet/pods/50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5/volumes"
Oct 30 12:53:43.758348 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:43.758309    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"]
Oct 30 12:53:48.749514 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:48.749466    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"]
Oct 30 12:53:48.762352 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:48.762278    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr" podUID="60d3c282-e813-4727-a259-719025376289" containerName="payload-processing" containerID="cri-o://d3e0843e6f22e1221370235e946fefb694c130a03ac475dbbfef967bc185e902" gracePeriod=25
Oct 30 12:53:48.769023 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:48.768985    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-98f58556-92xw4"]
Oct 30 12:53:48.958405 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:48.958364    4993 generic.go:356] "Generic (PLEG): container finished" podID="60d3c282-e813-4727-a259-719025376289" containerID="d3e0843e6f22e1221370235e946fefb694c130a03ac475dbbfef967bc185e902" exitCode=0
Oct 30 12:53:48.958852 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:48.958672    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr" event={"ID":"60d3c282-e813-4727-a259-719025376289","Type":"ContainerDied","Data":"d3e0843e6f22e1221370235e946fefb694c130a03ac475dbbfef967bc185e902"}
Oct 30 12:53:48.978902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:48.978868    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"]
Oct 30 12:53:48.979438 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:48.979378    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy" containerID="cri-o://0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1" gracePeriod=30
Oct 30 12:53:49.196879 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.196833    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:49.333511 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.333468    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d3c282-e813-4727-a259-719025376289-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d3c282-e813-4727-a259-719025376289-plugins-config-volume\") pod \"60d3c282-e813-4727-a259-719025376289\" (UID: \"60d3c282-e813-4727-a259-719025376289\") "
Oct 30 12:53:49.333735 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.333539    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d3c282-e813-4727-a259-719025376289-kube-api-access-gf2xn\" (UniqueName: \"kubernetes.io/projected/60d3c282-e813-4727-a259-719025376289-kube-api-access-gf2xn\") pod \"60d3c282-e813-4727-a259-719025376289\" (UID: \"60d3c282-e813-4727-a259-719025376289\") "
Oct 30 12:53:49.333967 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.333942    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d3c282-e813-4727-a259-719025376289-plugins-config-volume" pod "60d3c282-e813-4727-a259-719025376289" (UID: "60d3c282-e813-4727-a259-719025376289"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:49.335858 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.335823    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d3c282-e813-4727-a259-719025376289-kube-api-access-gf2xn" pod "60d3c282-e813-4727-a259-719025376289" (UID: "60d3c282-e813-4727-a259-719025376289"). InnerVolumeSpecName "kube-api-access-gf2xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:49.435100 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.435001    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d3c282-e813-4727-a259-719025376289-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:49.435100 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.435039    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2xn\" (UniqueName: \"kubernetes.io/projected/60d3c282-e813-4727-a259-719025376289-kube-api-access-gf2xn\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:49.695702 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.695613    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d241014-9c85-4f86-98d7-9d08cc91826f" path="/var/lib/kubelet/pods/8d241014-9c85-4f86-98d7-9d08cc91826f/volumes"
Oct 30 12:53:49.980305 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.980260    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr" event={"ID":"60d3c282-e813-4727-a259-719025376289","Type":"ContainerDied","Data":"4da29d41948f7ceffd00ccd4b8607bb0a49a11705fb0f70a30c6a1c4c2e7acc9"}
Oct 30 12:53:49.980838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.980337    4993 scope.go:122] "RemoveContainer" containerID="d3e0843e6f22e1221370235e946fefb694c130a03ac475dbbfef967bc185e902"
Oct 30 12:53:49.980838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:49.980281    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"
Oct 30 12:53:50.039294 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.039254    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"]
Oct 30 12:53:50.071773 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.071731    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-98f58556-f9d6f6b84-pkxqr"]
Oct 30 12:53:50.260743 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.260641    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"]
Oct 30 12:53:50.270233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.261724    4993 cpu_manager.go:414] "containerMap: removing container" podUID="60d3c282-e813-4727-a259-719025376289" containerName="payload-processing"
Oct 30 12:53:50.270473 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.270443    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d3c282-e813-4727-a259-719025376289" containerName="payload-processing"
Oct 30 12:53:50.270626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.270614    4993 cpu_manager.go:414] "containerMap: removing container" podUID="50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5" containerName="payload-pre-processing"
Oct 30 12:53:50.270700 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.270689    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5" containerName="payload-pre-processing"
Oct 30 12:53:50.270963 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.270949    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c7d0c4-3ac0-4b91-975a-876c4e8ba8b5" containerName="payload-pre-processing"
Oct 30 12:53:50.271070 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.271060    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d3c282-e813-4727-a259-719025376289" containerName="payload-processing"
Oct 30 12:53:50.275039 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.275009    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.293279 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.293189    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6ff0e6-openshift-default-dockercfg-xpjcg\""
Oct 30 12:53:50.301315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.301280    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"]
Oct 30 12:53:50.343985 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.343941    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.343985 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.343986    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.344250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.344143    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.344250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.344193    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd38c30-f926-4a99-93fb-47b73375ef5e-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.344331 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.344292    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.344401 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.344384    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqjzs\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-kube-api-access-zqjzs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.344467 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.344440    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.344595 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.344506    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.344595 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.344552    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.445475 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445415    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqjzs\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-kube-api-access-zqjzs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.445664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445562    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.445664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445604    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.445664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445660    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.445860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445707    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.445860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445733    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.445860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445804    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.445860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445842    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd38c30-f926-4a99-93fb-47b73375ef5e-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.446059 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.445880    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.447768 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.446779    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.447768 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.447050    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.447768 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.447161    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.447768 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.447723    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd38c30-f926-4a99-93fb-47b73375ef5e-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.448474 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.448450    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.462717 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.462618    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.463342 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.463276    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.478219 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.478127    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqjzs\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-kube-api-access-zqjzs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.478861 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.478837    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.614591 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.614435    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:50.982825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.982303    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"]
Oct 30 12:53:50.984778 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:50.984731    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd38c30_f926_4a99_93fb_47b73375ef5e.slice/crio-b568aa7495f017f4e59e4e57977d06bd9a29735f80e838b9a2cf5e615f27688c WatchSource:0}: Error finding container b568aa7495f017f4e59e4e57977d06bd9a29735f80e838b9a2cf5e615f27688c: Status 404 returned error can't find the container with id b568aa7495f017f4e59e4e57977d06bd9a29735f80e838b9a2cf5e615f27688c
Oct 30 12:53:50.986938 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.986904    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:50.987060 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.986972    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:50.987060 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:50.987004    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:51.691252 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:51.691198    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d3c282-e813-4727-a259-719025376289" path="/var/lib/kubelet/pods/60d3c282-e813-4727-a259-719025376289/volumes"
Oct 30 12:53:51.947776 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:51.947694    4993 patch_prober.go:29] interesting pod/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:53:51.947776 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:51.947750    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:53:51.998780 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:51.998721    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" event={"ID":"dbd38c30-f926-4a99-93fb-47b73375ef5e","Type":"ContainerStarted","Data":"81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2"}
Oct 30 12:53:51.999325 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:51.998795    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" event={"ID":"dbd38c30-f926-4a99-93fb-47b73375ef5e","Type":"ContainerStarted","Data":"b568aa7495f017f4e59e4e57977d06bd9a29735f80e838b9a2cf5e615f27688c"}
Oct 30 12:53:52.077284 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:52.074459    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" podStartSLOduration=2.074435818 podStartE2EDuration="2.074435818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:52.071766564 +0000 UTC m=+1296.547486206" watchObservedRunningTime="2026-10-30 12:53:52.074435818 +0000 UTC m=+1296.550155428"
Oct 30 12:53:52.615120 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:52.615079    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:52.621371 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:52.621339    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:53.005129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:53.005090    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:53.006038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:53.006018    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:53:54.384390 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.384348    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:54.514699 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.514658    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-podinfo\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.515091 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515072    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-certs\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.515155 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515100    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-kube-api-access-r8qt4\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-kube-api-access-r8qt4\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.515155 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515137    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-credential-socket\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.515304 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515166    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-socket\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.515451 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515433    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-socket" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:54.515570 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515557    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istiod-ca-cert\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.515632 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515610    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-data\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.515733 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515716    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-envoy\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.515889 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515856    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-credential-socket" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:54.515950 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515879    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-data" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:54.516005 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.515972    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-token\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-token\") pod \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\" (UID: \"ee23bbfd-0a30-485b-a94e-d959d94a6c47\") "
Oct 30 12:53:54.516140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.516115    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istiod-ca-cert" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:54.516462 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.516444    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:54.516462 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.516461    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:54.516626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.516470    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:54.516626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.516480    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:54.516626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.516481    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-certs" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:54.518221 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.518152    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-envoy" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:54.518943 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.518902    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-kube-api-access-r8qt4" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "kube-api-access-r8qt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:54.519082 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.519058    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-token" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:54.519264 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.519246    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-podinfo" pod "ee23bbfd-0a30-485b-a94e-d959d94a6c47" (UID: "ee23bbfd-0a30-485b-a94e-d959d94a6c47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:53:54.617931 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.617830    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:54.617931 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.617867    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:54.617931 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.617877    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:54.617931 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.617885    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8qt4\" (UniqueName: \"kubernetes.io/projected/ee23bbfd-0a30-485b-a94e-d959d94a6c47-kube-api-access-r8qt4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:54.617931 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:54.617894    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23bbfd-0a30-485b-a94e-d959d94a6c47-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:53:55.024855 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.024817    4993 generic.go:356] "Generic (PLEG): container finished" podID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerID="0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1" exitCode=0
Oct 30 12:53:55.030228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.025799    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"
Oct 30 12:53:55.030228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.027510    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" event={"ID":"ee23bbfd-0a30-485b-a94e-d959d94a6c47","Type":"ContainerDied","Data":"0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1"}
Oct 30 12:53:55.030228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.027571    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd" event={"ID":"ee23bbfd-0a30-485b-a94e-d959d94a6c47","Type":"ContainerDied","Data":"b31bdec95d58887926b0273522422d19917cf3aeebbb718ea2b8f9e3431919cd"}
Oct 30 12:53:55.030228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.027603    4993 scope.go:122] "RemoveContainer" containerID="0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1"
Oct 30 12:53:55.044480 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.044419    4993 scope.go:122] "RemoveContainer" containerID="8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223"
Oct 30 12:53:55.057506 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.057476    4993 scope.go:122] "RemoveContainer" containerID="0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1"
Oct 30 12:53:55.058012 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:53:55.057979    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1\": container with ID starting with 0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1 not found: ID does not exist" containerID="0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1"
Oct 30 12:53:55.058101 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.058041    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1"} err="failed to get container status \"0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1\": rpc error: code = NotFound desc = could not find container \"0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1\": container with ID starting with 0ef41e63f827abbc2a1b13416cdd7ee00ea066a54e05b8c5e67104540fde40c1 not found: ID does not exist"
Oct 30 12:53:55.058101 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.058069    4993 scope.go:122] "RemoveContainer" containerID="8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223"
Oct 30 12:53:55.059070 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:53:55.058517    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223\": container with ID starting with 8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223 not found: ID does not exist" containerID="8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223"
Oct 30 12:53:55.059070 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.058551    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223"} err="failed to get container status \"8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223\": rpc error: code = NotFound desc = could not find container \"8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223\": container with ID starting with 8c5041a5892f93930c24432435e182f43f48d731c2297bcf17c0e6ba7841c223 not found: ID does not exist"
Oct 30 12:53:55.104319 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.104280    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"]
Oct 30 12:53:55.114921 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.114882    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-98f58556-openshift-default-578678f5b9-9q7pd"]
Oct 30 12:53:55.694253 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:55.694125    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" path="/var/lib/kubelet/pods/ee23bbfd-0a30-485b-a94e-d959d94a6c47/volumes"
Oct 30 12:53:56.746508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.746415    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"]
Oct 30 12:53:56.747077 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.747052    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy"
Oct 30 12:53:56.747077 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.747082    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy"
Oct 30 12:53:56.747286 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.747277    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy"
Oct 30 12:53:56.747347 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.747296    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy"
Oct 30 12:53:56.751219 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.751178    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.774639 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.774599    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"]
Oct 30 12:53:56.785456 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.785412    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"kube-root-ca.crt\""
Oct 30 12:53:56.785456 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.785412    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"simulator-w3-551dd1-kserve-self-signed-certs\""
Oct 30 12:53:56.785803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.785566    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"default-dockercfg-mxsm5\""
Oct 30 12:53:56.785803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.785591    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"openshift-service-ca.crt\""
Oct 30 12:53:56.841842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.841802    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-home\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.842056 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.841993    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-tmp-dir\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.842202 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.842068    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b70714-bfd7-4054-8f7f-21849ff3d386-tls-certs\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.842202 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.842171    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-dshm\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.842356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.842271    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-model-cache\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.842356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.842322    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbh7w\" (UniqueName: \"kubernetes.io/projected/28b70714-bfd7-4054-8f7f-21849ff3d386-kube-api-access-hbh7w\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943123 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943081    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-home\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943142    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-tmp-dir\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943160    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b70714-bfd7-4054-8f7f-21849ff3d386-tls-certs\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943199    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-dshm\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943276    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-model-cache\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943303    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbh7w\" (UniqueName: \"kubernetes.io/projected/28b70714-bfd7-4054-8f7f-21849ff3d386-kube-api-access-hbh7w\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943641 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943619    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-tmp-dir\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943715 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943660    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-home\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.943814 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.943785    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-model-cache\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.945619 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.945596    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-dshm\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.945964 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.945944    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b70714-bfd7-4054-8f7f-21849ff3d386-tls-certs\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:56.972066 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:56.972019    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbh7w\" (UniqueName: \"kubernetes.io/projected/28b70714-bfd7-4054-8f7f-21849ff3d386-kube-api-access-hbh7w\") pod \"simulator-w3-551dd1-kserve-5948746798-drxvz\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") " pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:57.069783 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:57.069676    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:53:57.393588 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:57.393558    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"]
Oct 30 12:53:57.396761 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:53:57.396729    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28b70714_bfd7_4054_8f7f_21849ff3d386.slice/crio-e0c0fe14def56737820f0fecef585f05b6b9b14e027cbe9058d5c072d55cd276 WatchSource:0}: Error finding container e0c0fe14def56737820f0fecef585f05b6b9b14e027cbe9058d5c072d55cd276: Status 404 returned error can't find the container with id e0c0fe14def56737820f0fecef585f05b6b9b14e027cbe9058d5c072d55cd276
Oct 30 12:53:58.048479 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:58.048436    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" event={"ID":"28b70714-bfd7-4054-8f7f-21849ff3d386","Type":"ContainerStarted","Data":"e0c0fe14def56737820f0fecef585f05b6b9b14e027cbe9058d5c072d55cd276"}
Oct 30 12:53:59.076086 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:59.076036    4993 generic.go:356] "Generic (PLEG): container finished" podID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerID="778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee" exitCode=0
Oct 30 12:53:59.076703 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:59.076116    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" event={"ID":"02d5d695-584f-40a5-92a0-eb12f0b5b959","Type":"ContainerDied","Data":"778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee"}
Oct 30 12:53:59.077241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:59.077192    4993 scope.go:122] "RemoveContainer" containerID="778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee"
Oct 30 12:53:59.079016 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:59.078935    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" event={"ID":"28b70714-bfd7-4054-8f7f-21849ff3d386","Type":"ContainerStarted","Data":"027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1"}
Oct 30 12:53:59.079284 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:59.079248    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:59.079358 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:59.079343    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:53:59.079420 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:53:59.079390    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:54:00.080409 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:00.080340    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:54:00.092147 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:00.092105    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" event={"ID":"02d5d695-584f-40a5-92a0-eb12f0b5b959","Type":"ContainerStarted","Data":"6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff"}
Oct 30 12:54:00.179409 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:00.160153    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podStartSLOduration=2.900048727 podStartE2EDuration="4.160128956s" totalImagesPullingTime="1.260080229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:59.189589451 +0000 UTC m=+1303.665309062" watchObservedRunningTime="2026-10-30 12:54:00.160128956 +0000 UTC m=+1304.635848573"
Oct 30 12:54:00.846230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:00.845595    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:54:00.846230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:00.845726    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:54:00.846230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:00.845769    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 12:54:01.845107 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:01.845067    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:54:01.845553 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:01.845122    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 12:54:02.844805 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:02.844766    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:54:02.845043 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:02.844828    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 12:54:03.844658 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:03.844621    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:54:03.845196 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:03.845156    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 12:54:04.844614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:04.844568    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:54:04.844797 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:04.844648    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 12:54:05.849088 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:05.849056    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:54:05.849633 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:05.849471    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:54:05.850315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:05.850295    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 12:54:07.070011 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:07.069964    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:54:07.083022 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:07.082987    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:54:07.163192 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:07.163144    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:54:07.984753 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:07.984715    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"]
Oct 30 12:54:07.985484 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:07.985459    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy"
Oct 30 12:54:07.985582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:07.985487    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee23bbfd-0a30-485b-a94e-d959d94a6c47" containerName="istio-proxy"
Oct 30 12:54:07.990031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:07.990004    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.006664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.005697    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"]
Oct 30 12:54:08.015712 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.015683    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"default-dockercfg-pdp25\""
Oct 30 12:54:08.015952 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.015930    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"simulator-w4-e652ca-kserve-self-signed-certs\""
Oct 30 12:54:08.020451 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.020418    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"kube-root-ca.crt\""
Oct 30 12:54:08.020581 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.020456    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"openshift-service-ca.crt\""
Oct 30 12:54:08.039062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.039022    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de7529f-c724-4629-b333-2a1eb259794d-tls-certs\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.039062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.039065    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9x8\" (UniqueName: \"kubernetes.io/projected/0de7529f-c724-4629-b333-2a1eb259794d-kube-api-access-vr9x8\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.039345 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.039236    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-home\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.039345 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.039331    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-tmp-dir\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.039453 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.039359    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-dshm\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.039453 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.039393    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-model-cache\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.140880 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.140847    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-dshm\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.141426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.141401    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-model-cache\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.141610 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.141589    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de7529f-c724-4629-b333-2a1eb259794d-tls-certs\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.141689 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.141628    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9x8\" (UniqueName: \"kubernetes.io/projected/0de7529f-c724-4629-b333-2a1eb259794d-kube-api-access-vr9x8\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.141744 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.141713    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-home\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.141798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.141775    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-tmp-dir\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.142228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.142192    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-tmp-dir\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.142949 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.142923    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-model-cache\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.143237 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.143197    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-home\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.145903 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.145874    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-dshm\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.146026 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.145882    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de7529f-c724-4629-b333-2a1eb259794d-tls-certs\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.168814 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.168784    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9x8\" (UniqueName: \"kubernetes.io/projected/0de7529f-c724-4629-b333-2a1eb259794d-kube-api-access-vr9x8\") pod \"simulator-w4-e652ca-kserve-b4664f99f-nz4nh\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") " pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.318123 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.315868    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:08.577767 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:54:08.577664    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:54:08.630390 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:08.630359    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"]
Oct 30 12:54:08.632037 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:54:08.631997    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0de7529f_c724_4629_b333_2a1eb259794d.slice/crio-59d6cad21d5fee2af5e7e7156bbefa149826a39c8268190d9664acf17b19624a WatchSource:0}: Error finding container 59d6cad21d5fee2af5e7e7156bbefa149826a39c8268190d9664acf17b19624a: Status 404 returned error can't find the container with id 59d6cad21d5fee2af5e7e7156bbefa149826a39c8268190d9664acf17b19624a
Oct 30 12:54:09.162961 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:09.162920    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" event={"ID":"0de7529f-c724-4629-b333-2a1eb259794d","Type":"ContainerStarted","Data":"94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425"}
Oct 30 12:54:09.163389 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:09.162977    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" event={"ID":"0de7529f-c724-4629-b333-2a1eb259794d","Type":"ContainerStarted","Data":"59d6cad21d5fee2af5e7e7156bbefa149826a39c8268190d9664acf17b19624a"}
Oct 30 12:54:09.212769 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:09.212699    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podStartSLOduration=2.212683454 podStartE2EDuration="2.212683454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:09.210672936 +0000 UTC m=+1313.686392546" watchObservedRunningTime="2026-10-30 12:54:09.212683454 +0000 UTC m=+1313.688403065"
Oct 30 12:54:09.317422 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:09.317384    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:13.528011 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.527972    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"]
Oct 30 12:54:13.550528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.550487    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.566609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.566568    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"]
Oct 30 12:54:13.572770 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.572737    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"openshift-service-ca.crt\""
Oct 30 12:54:13.603231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.601479    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"kube-root-ca.crt\""
Oct 30 12:54:13.603231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.601632    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"simulator-w0-7a8b03-kserve-self-signed-certs\""
Oct 30 12:54:13.603231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.601762    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"default-dockercfg-sqhgq\""
Oct 30 12:54:13.637337 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.637283    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tmp-dir\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.637693 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.637660    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-model-cache\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.637914 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.637895    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tls-certs\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.638137 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.638117    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpj4b\" (UniqueName: \"kubernetes.io/projected/7efd7f65-f9e5-489c-b010-1ce9b04311eb-kube-api-access-bpj4b\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.638381 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.638314    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-home\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.638552 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.638527    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-dshm\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.740228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.739758    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpj4b\" (UniqueName: \"kubernetes.io/projected/7efd7f65-f9e5-489c-b010-1ce9b04311eb-kube-api-access-bpj4b\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.740228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.739854    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-home\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.740228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.739902    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-dshm\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.740228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.740042    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tmp-dir\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.740228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.740091    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-model-cache\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.740228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.740126    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tls-certs\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.741408 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.741048    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-home\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.741408 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.741365    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-model-cache\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.741870 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.741851    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tmp-dir\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.744014 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.743991    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tls-certs\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.744184 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.744163    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-dshm\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.781295 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.775140    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpj4b\" (UniqueName: \"kubernetes.io/projected/7efd7f65-f9e5-489c-b010-1ce9b04311eb-kube-api-access-bpj4b\") pod \"simulator-w0-7a8b03-kserve-5dc64556f4-fqpph\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") " pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:13.943725 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:13.943689    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:14.294564 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:14.294534    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"]
Oct 30 12:54:14.298271 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:54:14.297228    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7efd7f65_f9e5_489c_b010_1ce9b04311eb.slice/crio-dcd53e179fcf04545b1fb8a8d120359816ab441b5c854103d713453f075f5beb WatchSource:0}: Error finding container dcd53e179fcf04545b1fb8a8d120359816ab441b5c854103d713453f075f5beb: Status 404 returned error can't find the container with id dcd53e179fcf04545b1fb8a8d120359816ab441b5c854103d713453f075f5beb
Oct 30 12:54:15.229158 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:15.229118    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" event={"ID":"7efd7f65-f9e5-489c-b010-1ce9b04311eb","Type":"ContainerStarted","Data":"a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052"}
Oct 30 12:54:15.229713 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:15.229165    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" event={"ID":"7efd7f65-f9e5-489c-b010-1ce9b04311eb","Type":"ContainerStarted","Data":"dcd53e179fcf04545b1fb8a8d120359816ab441b5c854103d713453f075f5beb"}
Oct 30 12:54:15.277178 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:15.277124    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podStartSLOduration=2.277108634 podStartE2EDuration="2.277108634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:15.274771155 +0000 UTC m=+1319.750490765" watchObservedRunningTime="2026-10-30 12:54:15.277108634 +0000 UTC m=+1319.752828244"
Oct 30 12:54:15.944693 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:15.944652    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:18.073495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.073456    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"]
Oct 30 12:54:18.079121 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.079094    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.099787 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.099746    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"]
Oct 30 12:54:18.109340 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.109304    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"kube-root-ca.crt\""
Oct 30 12:54:18.109581 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.109303    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"openshift-service-ca.crt\""
Oct 30 12:54:18.109700 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.109451    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"simulator-w2-bb86fd-kserve-self-signed-certs\""
Oct 30 12:54:18.110077 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.110045    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"default-dockercfg-5xdjs\""
Oct 30 12:54:18.205711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.205660    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tmp-dir\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.205711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.205705    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-model-cache\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.205998 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.205738    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-dshm\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.205998 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.205920    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tls-certs\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.205998 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.205991    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-home\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.206189 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.206073    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cw8\" (UniqueName: \"kubernetes.io/projected/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-kube-api-access-p5cw8\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.307308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.307253    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tls-certs\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.307515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.307340    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-home\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.307515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.307435    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cw8\" (UniqueName: \"kubernetes.io/projected/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-kube-api-access-p5cw8\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.307671 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.307547    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tmp-dir\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.307671 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.307581    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-model-cache\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.307671 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.307629    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-dshm\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.309336 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.309267    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-home\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.309541 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.309368    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-model-cache\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.309939 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.309916    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tmp-dir\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.312278 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.312248    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tls-certs\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.320759 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.317832    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:18.328751 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.328662    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-dshm\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.334316 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.334273    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cw8\" (UniqueName: \"kubernetes.io/projected/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-kube-api-access-p5cw8\") pod \"simulator-w2-bb86fd-kserve-556c6ddcc-kck9t\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") " pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.335732 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.335707    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:18.396362 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.396321    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:18.722378 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:18.722299    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"]
Oct 30 12:54:18.750583 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:54:18.750525    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f9baa7e_a484_4d2e_aeb1_238028c1a77f.slice/crio-433e6e47e8b4c60be345d4c20ff3463804518088646bae7e3ef710a8c2c1a3ef WatchSource:0}: Error finding container 433e6e47e8b4c60be345d4c20ff3463804518088646bae7e3ef710a8c2c1a3ef: Status 404 returned error can't find the container with id 433e6e47e8b4c60be345d4c20ff3463804518088646bae7e3ef710a8c2c1a3ef
Oct 30 12:54:19.273511 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.273477    4993 generic.go:356] "Generic (PLEG): container finished" podID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerID="81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2" exitCode=0
Oct 30 12:54:19.274014 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.273562    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" event={"ID":"dbd38c30-f926-4a99-93fb-47b73375ef5e","Type":"ContainerDied","Data":"81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2"}
Oct 30 12:54:19.274082 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.274050    4993 scope.go:122] "RemoveContainer" containerID="81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2"
Oct 30 12:54:19.275627 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.275598    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" event={"ID":"1f9baa7e-a484-4d2e-aeb1-238028c1a77f","Type":"ContainerStarted","Data":"447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe"}
Oct 30 12:54:19.275760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.275638    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" event={"ID":"1f9baa7e-a484-4d2e-aeb1-238028c1a77f","Type":"ContainerStarted","Data":"433e6e47e8b4c60be345d4c20ff3463804518088646bae7e3ef710a8c2c1a3ef"}
Oct 30 12:54:19.275760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.275667    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:54:19.275760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.275737    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:54:19.275933 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.275780    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:54:19.287754 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.287727    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:54:19.396857 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.396820    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:19.504108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:19.504039    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podStartSLOduration=1.50402273 podStartE2EDuration="1.50402273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:19.501301334 +0000 UTC m=+1323.977020946" watchObservedRunningTime="2026-10-30 12:54:19.50402273 +0000 UTC m=+1323.979742340"
Oct 30 12:54:20.289417 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:20.289376    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" event={"ID":"dbd38c30-f926-4a99-93fb-47b73375ef5e","Type":"ContainerStarted","Data":"304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0"}
Oct 30 12:54:20.615059 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:20.614965    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:54:20.616683 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:20.616651    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:54:20.616778 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:20.616730    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 12:54:21.615330 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:21.615293    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:54:21.615741 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:21.615350    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 12:54:22.614927 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:22.614890    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:54:22.615192 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:22.615161    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 12:54:23.005839 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:23.005789    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:54:23.614933 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:23.614892    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:54:23.615104 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:23.614959    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 12:54:23.944830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:23.944710    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:23.985233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:23.984635    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:24.349984 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:24.349950    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:54:24.615708 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:24.615622    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:54:24.615708 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:24.615679    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 12:54:25.619626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:25.619587    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:54:25.620754 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:25.620730    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 12:54:28.396850 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:28.396800    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:28.409785 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:28.409749    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:29.386062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:29.386028    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:54:38.572355 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:54:38.572307    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:54:43.399073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.399039    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"]
Oct 30 12:54:43.403066 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.403043    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.427301 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.427267    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"]
Oct 30 12:54:43.438627 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.438592    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"premium-w0-7a8b03-kserve-self-signed-certs\""
Oct 30 12:54:43.545066 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.545028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-home\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.545066 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.545068    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7m7g\" (UniqueName: \"kubernetes.io/projected/0769aaca-8e70-4393-a400-5519741bc5e6-kube-api-access-b7m7g\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.545309 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.545163    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-tmp-dir\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.545309 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.545227    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-model-cache\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.545376 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.545326    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-dshm\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.545412 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.545388    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0769aaca-8e70-4393-a400-5519741bc5e6-tls-certs\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646031    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-dshm\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646334 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646087    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0769aaca-8e70-4393-a400-5519741bc5e6-tls-certs\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646334 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646253    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-home\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646334 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646295    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7m7g\" (UniqueName: \"kubernetes.io/projected/0769aaca-8e70-4393-a400-5519741bc5e6-kube-api-access-b7m7g\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646517 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646368    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-tmp-dir\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646517 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646420    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-model-cache\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646667 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646647    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-home\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646799    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-model-cache\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.646955 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.646929    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-tmp-dir\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.652187 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.652105    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0769aaca-8e70-4393-a400-5519741bc5e6-tls-certs\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.653627 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.653594    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-dshm\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.680497 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.680461    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7m7g\" (UniqueName: \"kubernetes.io/projected/0769aaca-8e70-4393-a400-5519741bc5e6-kube-api-access-b7m7g\") pod \"premium-w0-7a8b03-kserve-69c789c955-4q7nw\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") " pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:43.718390 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:43.718351    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:44.033939 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:44.033906    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"]
Oct 30 12:54:44.035475 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:54:44.035442    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0769aaca_8e70_4393_a400_5519741bc5e6.slice/crio-622ff6208a3f2681cd87bfd60ef41e966cf90d9e0197613201e2e944e88e36fe WatchSource:0}: Error finding container 622ff6208a3f2681cd87bfd60ef41e966cf90d9e0197613201e2e944e88e36fe: Status 404 returned error can't find the container with id 622ff6208a3f2681cd87bfd60ef41e966cf90d9e0197613201e2e944e88e36fe
Oct 30 12:54:44.498486 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:44.498447    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" event={"ID":"0769aaca-8e70-4393-a400-5519741bc5e6","Type":"ContainerStarted","Data":"015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c"}
Oct 30 12:54:44.498897 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:44.498496    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" event={"ID":"0769aaca-8e70-4393-a400-5519741bc5e6","Type":"ContainerStarted","Data":"622ff6208a3f2681cd87bfd60ef41e966cf90d9e0197613201e2e944e88e36fe"}
Oct 30 12:54:44.545003 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:44.544940    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podStartSLOduration=1.544926477 podStartE2EDuration="1.544926477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:44.543363764 +0000 UTC m=+1349.019083375" watchObservedRunningTime="2026-10-30 12:54:44.544926477 +0000 UTC m=+1349.020646086"
Oct 30 12:54:44.719175 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:44.719138    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:53.718653 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:53.718604    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:53.732272 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:53.732196    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:54:54.584433 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:54:54.584392    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:55:08.576159 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:55:08.576113    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:55:25.051554 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.051508    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"]
Oct 30 12:55:25.056036 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.056005    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.078424 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.078390    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb86fd-openshift-default-dockercfg-bqgfs\""
Oct 30 12:55:25.093856 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.093816    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"]
Oct 30 12:55:25.146057 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146007    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.146277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146071    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-kube-api-access-xvcwm\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.146277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146171    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.146277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146246    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.146460 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146310    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.146460 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146365    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.146460 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146419    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.147040 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146467    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.147040 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.146500    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.247833 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.247795    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.247882    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.247915    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.247971    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248306 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248062    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248306 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248116    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248306 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248170    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-kube-api-access-xvcwm\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248306 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248235    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248516 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248401    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248736 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248711    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248891 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248797    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.248992 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248945    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.249080 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.248988    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.250935 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.250911    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.250935 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.250924    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.275977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.275942    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.276149 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.276064    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-kube-api-access-xvcwm\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.370004 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.369913    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:25.711907 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.703899    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"]
Oct 30 12:55:25.717407 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.717368    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:55:25.717536 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.717459    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:55:25.717536 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.717503    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:55:25.805039 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.804997    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" event={"ID":"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed","Type":"ContainerStarted","Data":"600e5d2e1c2c3b236116ddc5d03781fdb73bc0c17c4999880f87af076b5bb734"}
Oct 30 12:55:25.805254 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.805056    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" event={"ID":"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed","Type":"ContainerStarted","Data":"99e0ca9ac752f3d17262ecc9f1d7b32c677f629d168930a39a423046117ea963"}
Oct 30 12:55:25.855962 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:25.855891    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podStartSLOduration=0.85587733 podStartE2EDuration="855.87733ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:25.854702801 +0000 UTC m=+1390.330422414" watchObservedRunningTime="2026-10-30 12:55:25.85587733 +0000 UTC m=+1390.331596939"
Oct 30 12:55:26.380294 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:26.379879    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:26.389688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:26.383404    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:55:26.389688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:26.383480    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 12:55:27.374241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:27.373780    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:55:27.374241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:27.373867    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 12:55:28.371231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.370623    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:55:28.371231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.370691    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 12:55:28.646532 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.646440    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"]
Oct 30 12:55:28.650452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.650429    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.675230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.669571    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:55:28.675230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.669762    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:55:28.687079 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.687043    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:55:28.687314 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.687164    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:55:28.689894 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.689857    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"]
Oct 30 12:55:28.711677 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.711633    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.711966 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.711946    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.712181 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.712165    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmcs\" (UniqueName: \"kubernetes.io/projected/0ac3e73d-c73a-4fb8-8473-ba398564b702-kube-api-access-lrmcs\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.813228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.813042    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.813228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.813102    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.813228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.813194    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmcs\" (UniqueName: \"kubernetes.io/projected/0ac3e73d-c73a-4fb8-8473-ba398564b702-kube-api-access-lrmcs\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.816234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.814370    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.852037 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.852000    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmcs\" (UniqueName: \"kubernetes.io/projected/0ac3e73d-c73a-4fb8-8473-ba398564b702-kube-api-access-lrmcs\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.861241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.858980    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a32dec81-fmtjt\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:28.995230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:28.994261    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:29.372660 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:29.372620    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:55:29.373320 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:29.373280    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 12:55:29.521949 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:29.521900    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"]
Oct 30 12:55:29.526384 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:55:29.526323    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac3e73d_c73a_4fb8_8473_ba398564b702.slice/crio-6d65a9d0ecdf5daa697d621ec17d662dae680619ff06ed691cc49451d72ccc51 WatchSource:0}: Error finding container 6d65a9d0ecdf5daa697d621ec17d662dae680619ff06ed691cc49451d72ccc51: Status 404 returned error can't find the container with id 6d65a9d0ecdf5daa697d621ec17d662dae680619ff06ed691cc49451d72ccc51
Oct 30 12:55:29.885009 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:29.884967    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt" event={"ID":"0ac3e73d-c73a-4fb8-8473-ba398564b702","Type":"ContainerStarted","Data":"6d65a9d0ecdf5daa697d621ec17d662dae680619ff06ed691cc49451d72ccc51"}
Oct 30 12:55:30.371232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:30.370624    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:55:30.371232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:30.370692    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 12:55:30.896377 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:30.896328    4993 generic.go:356] "Generic (PLEG): container finished" podID="0ac3e73d-c73a-4fb8-8473-ba398564b702" containerID="a3344f60f8d6a56e1f306b13b1e4d3470cc328ff5a4e4d08ece688552e97f755" exitCode=0
Oct 30 12:55:30.896902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:30.896470    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt" event={"ID":"0ac3e73d-c73a-4fb8-8473-ba398564b702","Type":"ContainerDied","Data":"a3344f60f8d6a56e1f306b13b1e4d3470cc328ff5a4e4d08ece688552e97f755"}
Oct 30 12:55:31.371232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:31.370663    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:55:31.371232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:31.370736    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 12:55:32.086789 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.086757    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:32.106665 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.106622    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-xks-ca\") pod \"0ac3e73d-c73a-4fb8-8473-ba398564b702\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") "
Oct 30 12:55:32.106844 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.106706    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac3e73d-c73a-4fb8-8473-ba398564b702-kube-api-access-lrmcs\" (UniqueName: \"kubernetes.io/projected/0ac3e73d-c73a-4fb8-8473-ba398564b702-kube-api-access-lrmcs\") pod \"0ac3e73d-c73a-4fb8-8473-ba398564b702\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") "
Oct 30 12:55:32.106844 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.106763    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-openshift-ca\") pod \"0ac3e73d-c73a-4fb8-8473-ba398564b702\" (UID: \"0ac3e73d-c73a-4fb8-8473-ba398564b702\") "
Oct 30 12:55:32.107175 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.107147    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-openshift-ca" pod "0ac3e73d-c73a-4fb8-8473-ba398564b702" (UID: "0ac3e73d-c73a-4fb8-8473-ba398564b702"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:32.108776 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.108747    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-xks-ca" pod "0ac3e73d-c73a-4fb8-8473-ba398564b702" (UID: "0ac3e73d-c73a-4fb8-8473-ba398564b702"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:32.109019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.108992    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac3e73d-c73a-4fb8-8473-ba398564b702-kube-api-access-lrmcs" pod "0ac3e73d-c73a-4fb8-8473-ba398564b702" (UID: "0ac3e73d-c73a-4fb8-8473-ba398564b702"). InnerVolumeSpecName "kube-api-access-lrmcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:32.208701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.208593    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:32.208701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.208644    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrmcs\" (UniqueName: \"kubernetes.io/projected/0ac3e73d-c73a-4fb8-8473-ba398564b702-kube-api-access-lrmcs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:32.208701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.208658    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac3e73d-c73a-4fb8-8473-ba398564b702-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:32.370469 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.370423    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:55:32.370657 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.370496    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 12:55:32.911035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.911004    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"
Oct 30 12:55:32.911276 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.911007    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt" event={"ID":"0ac3e73d-c73a-4fb8-8473-ba398564b702","Type":"ContainerDied","Data":"6d65a9d0ecdf5daa697d621ec17d662dae680619ff06ed691cc49451d72ccc51"}
Oct 30 12:55:32.911276 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:32.911135    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d65a9d0ecdf5daa697d621ec17d662dae680619ff06ed691cc49451d72ccc51"
Oct 30 12:55:33.376698 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:33.376663    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:33.377231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:33.376946    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:33.377998 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:33.377978    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 12:55:35.744789 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.744750    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"]
Oct 30 12:55:35.745236 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.745221    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0ac3e73d-c73a-4fb8-8473-ba398564b702" containerName="revoke-keys"
Oct 30 12:55:35.745299 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.745237    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac3e73d-c73a-4fb8-8473-ba398564b702" containerName="revoke-keys"
Oct 30 12:55:35.745335 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.745319    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac3e73d-c73a-4fb8-8473-ba398564b702" containerName="revoke-keys"
Oct 30 12:55:35.748622 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.748601    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.765416 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.765385    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:55:35.765579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.765383    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:55:35.779542 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.779488    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:55:35.779542 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.779510    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:55:35.782688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.782659    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"]
Oct 30 12:55:35.847288 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.847250    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf73034-530a-4278-8401-4b9700610c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.847451 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.847411    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnpg2\" (UniqueName: \"kubernetes.io/projected/baf73034-530a-4278-8401-4b9700610c32-kube-api-access-mnpg2\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.847513 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.847472    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf73034-530a-4278-8401-4b9700610c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.948388 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.948347    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnpg2\" (UniqueName: \"kubernetes.io/projected/baf73034-530a-4278-8401-4b9700610c32-kube-api-access-mnpg2\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.948583 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.948400    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf73034-530a-4278-8401-4b9700610c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.948583 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.948452    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf73034-530a-4278-8401-4b9700610c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.949149 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.949126    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf73034-530a-4278-8401-4b9700610c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.950976 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.950952    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf73034-530a-4278-8401-4b9700610c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:35.974092 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:35.974060    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnpg2\" (UniqueName: \"kubernetes.io/projected/baf73034-530a-4278-8401-4b9700610c32-kube-api-access-mnpg2\") pod \"maas-api-revoke-sub-b09ffb25-z6gcg\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:36.062035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:36.061939    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:36.369005 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:36.368960    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"]
Oct 30 12:55:36.371691 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:55:36.371646    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf73034_530a_4278_8401_4b9700610c32.slice/crio-ac82b3102a03be2c79d4030deaf65ea7a373b61580c45a570552009bfd36fa2b WatchSource:0}: Error finding container ac82b3102a03be2c79d4030deaf65ea7a373b61580c45a570552009bfd36fa2b: Status 404 returned error can't find the container with id ac82b3102a03be2c79d4030deaf65ea7a373b61580c45a570552009bfd36fa2b
Oct 30 12:55:36.936910 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:36.936818    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg" event={"ID":"baf73034-530a-4278-8401-4b9700610c32","Type":"ContainerStarted","Data":"9140b26eed58b0d9e2c7722b78de45fb62509d2d2058da94a04993a046b359a7"}
Oct 30 12:55:36.936910 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:36.936873    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg" event={"ID":"baf73034-530a-4278-8401-4b9700610c32","Type":"ContainerStarted","Data":"ac82b3102a03be2c79d4030deaf65ea7a373b61580c45a570552009bfd36fa2b"}
Oct 30 12:55:36.991889 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:36.991833    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg" podStartSLOduration=1.991819132 podStartE2EDuration="1.991819132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:36.989021548 +0000 UTC m=+1401.464741158" watchObservedRunningTime="2026-10-30 12:55:36.991819132 +0000 UTC m=+1401.467538741"
Oct 30 12:55:37.944452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:37.944418    4993 generic.go:356] "Generic (PLEG): container finished" podID="baf73034-530a-4278-8401-4b9700610c32" containerID="9140b26eed58b0d9e2c7722b78de45fb62509d2d2058da94a04993a046b359a7" exitCode=0
Oct 30 12:55:37.944854 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:37.944510    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg" event={"ID":"baf73034-530a-4278-8401-4b9700610c32","Type":"ContainerDied","Data":"9140b26eed58b0d9e2c7722b78de45fb62509d2d2058da94a04993a046b359a7"}
Oct 30 12:55:38.573233 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:55:38.573170    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:55:39.114819 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.114790    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:39.184016 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.183974    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf73034-530a-4278-8401-4b9700610c32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf73034-530a-4278-8401-4b9700610c32-maas-api-openshift-ca\") pod \"baf73034-530a-4278-8401-4b9700610c32\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") "
Oct 30 12:55:39.184016 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.184019    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf73034-530a-4278-8401-4b9700610c32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf73034-530a-4278-8401-4b9700610c32-maas-api-xks-ca\") pod \"baf73034-530a-4278-8401-4b9700610c32\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") "
Oct 30 12:55:39.184325 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.184080    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf73034-530a-4278-8401-4b9700610c32-kube-api-access-mnpg2\" (UniqueName: \"kubernetes.io/projected/baf73034-530a-4278-8401-4b9700610c32-kube-api-access-mnpg2\") pod \"baf73034-530a-4278-8401-4b9700610c32\" (UID: \"baf73034-530a-4278-8401-4b9700610c32\") "
Oct 30 12:55:39.184454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.184423    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf73034-530a-4278-8401-4b9700610c32-maas-api-openshift-ca" pod "baf73034-530a-4278-8401-4b9700610c32" (UID: "baf73034-530a-4278-8401-4b9700610c32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:39.186465 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.186433    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf73034-530a-4278-8401-4b9700610c32-maas-api-xks-ca" pod "baf73034-530a-4278-8401-4b9700610c32" (UID: "baf73034-530a-4278-8401-4b9700610c32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:39.186578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.186475    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf73034-530a-4278-8401-4b9700610c32-kube-api-access-mnpg2" pod "baf73034-530a-4278-8401-4b9700610c32" (UID: "baf73034-530a-4278-8401-4b9700610c32"). InnerVolumeSpecName "kube-api-access-mnpg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:39.285570 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.285525    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf73034-530a-4278-8401-4b9700610c32-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:39.285570 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.285562    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf73034-530a-4278-8401-4b9700610c32-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:39.285570 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.285573    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnpg2\" (UniqueName: \"kubernetes.io/projected/baf73034-530a-4278-8401-4b9700610c32-kube-api-access-mnpg2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:39.959760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.959666    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg" event={"ID":"baf73034-530a-4278-8401-4b9700610c32","Type":"ContainerDied","Data":"ac82b3102a03be2c79d4030deaf65ea7a373b61580c45a570552009bfd36fa2b"}
Oct 30 12:55:39.959760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.959726    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac82b3102a03be2c79d4030deaf65ea7a373b61580c45a570552009bfd36fa2b"
Oct 30 12:55:39.959760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:39.959685    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"
Oct 30 12:55:43.806489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.806451    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"]
Oct 30 12:55:43.806917 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.806903    4993 cpu_manager.go:414] "containerMap: removing container" podUID="baf73034-530a-4278-8401-4b9700610c32" containerName="revoke-keys"
Oct 30 12:55:43.806955 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.806918    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf73034-530a-4278-8401-4b9700610c32" containerName="revoke-keys"
Oct 30 12:55:43.807008 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.806999    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf73034-530a-4278-8401-4b9700610c32" containerName="revoke-keys"
Oct 30 12:55:43.810481 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.810453    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.831710 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.831672    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"]
Oct 30 12:55:43.838341 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.838304    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.838559 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.838479    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.838559 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.838525    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdf2v\" (UniqueName: \"kubernetes.io/projected/a9dc4580-c625-4505-950d-e7579d3d2af9-kube-api-access-sdf2v\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.841764 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.841723    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:55:43.841764 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.841739    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:55:43.841994 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.841977    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:55:43.842059 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.842022    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:55:43.940066 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.940032    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.940281 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.940090    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.940281 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.940257    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdf2v\" (UniqueName: \"kubernetes.io/projected/a9dc4580-c625-4505-950d-e7579d3d2af9-kube-api-access-sdf2v\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.940956 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.940932    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.942550 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.942530    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:43.965894 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:43.965861    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdf2v\" (UniqueName: \"kubernetes.io/projected/a9dc4580-c625-4505-950d-e7579d3d2af9-kube-api-access-sdf2v\") pod \"maas-api-revoke-sub-13d9f9eb-r8jcl\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:44.125827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:44.125734    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:44.447725 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:44.447693    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"]
Oct 30 12:55:45.004054 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:45.004006    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl" event={"ID":"a9dc4580-c625-4505-950d-e7579d3d2af9","Type":"ContainerStarted","Data":"39476e280fb356178ae87d7d023f1494589e0e5996a3816e583d1f0a2fda4e45"}
Oct 30 12:55:45.004482 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:45.004073    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl" event={"ID":"a9dc4580-c625-4505-950d-e7579d3d2af9","Type":"ContainerStarted","Data":"440f48f49f80870b6cd267f9464340d1a113b44a038bd96c25bc5a39af1d9435"}
Oct 30 12:55:45.053422 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:45.053297    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl" podStartSLOduration=2.053280392 podStartE2EDuration="2.053280392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:45.050624755 +0000 UTC m=+1409.526344366" watchObservedRunningTime="2026-10-30 12:55:45.053280392 +0000 UTC m=+1409.529000095"
Oct 30 12:55:46.012005 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.011913    4993 generic.go:356] "Generic (PLEG): container finished" podID="a9dc4580-c625-4505-950d-e7579d3d2af9" containerID="39476e280fb356178ae87d7d023f1494589e0e5996a3816e583d1f0a2fda4e45" exitCode=0
Oct 30 12:55:46.012539 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.012009    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl" event={"ID":"a9dc4580-c625-4505-950d-e7579d3d2af9","Type":"ContainerDied","Data":"39476e280fb356178ae87d7d023f1494589e0e5996a3816e583d1f0a2fda4e45"}
Oct 30 12:55:46.739382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.739340    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"]
Oct 30 12:55:46.745572 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.745541    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.761933 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.761901    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"]
Oct 30 12:55:46.773937 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.773903    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.774120 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.773962    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8f2\" (UniqueName: \"kubernetes.io/projected/43afe350-83fe-4157-9ebc-1f36b9a9d008-kube-api-access-9x8f2\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.774183 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.774119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.875244 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.875198    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.875487 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.875319    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.875487 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.875354    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8f2\" (UniqueName: \"kubernetes.io/projected/43afe350-83fe-4157-9ebc-1f36b9a9d008-kube-api-access-9x8f2\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.876054 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.876029    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.877773 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.877750    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:46.900724 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:46.900693    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8f2\" (UniqueName: \"kubernetes.io/projected/43afe350-83fe-4157-9ebc-1f36b9a9d008-kube-api-access-9x8f2\") pod \"maas-api-revoke-sub-16434684-2vvqc\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:47.058753 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.058712    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:47.203228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.203140    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:47.280285 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.280173    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-openshift-ca\") pod \"a9dc4580-c625-4505-950d-e7579d3d2af9\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") "
Oct 30 12:55:47.280463 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.280358    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-xks-ca\") pod \"a9dc4580-c625-4505-950d-e7579d3d2af9\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") "
Oct 30 12:55:47.280534 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.280471    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9dc4580-c625-4505-950d-e7579d3d2af9-kube-api-access-sdf2v\" (UniqueName: \"kubernetes.io/projected/a9dc4580-c625-4505-950d-e7579d3d2af9-kube-api-access-sdf2v\") pod \"a9dc4580-c625-4505-950d-e7579d3d2af9\" (UID: \"a9dc4580-c625-4505-950d-e7579d3d2af9\") "
Oct 30 12:55:47.281853 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.281805    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-openshift-ca" pod "a9dc4580-c625-4505-950d-e7579d3d2af9" (UID: "a9dc4580-c625-4505-950d-e7579d3d2af9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:47.283603 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.283576    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dc4580-c625-4505-950d-e7579d3d2af9-kube-api-access-sdf2v" pod "a9dc4580-c625-4505-950d-e7579d3d2af9" (UID: "a9dc4580-c625-4505-950d-e7579d3d2af9"). InnerVolumeSpecName "kube-api-access-sdf2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:47.284646 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.284611    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-xks-ca" pod "a9dc4580-c625-4505-950d-e7579d3d2af9" (UID: "a9dc4580-c625-4505-950d-e7579d3d2af9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:47.382432 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.382390    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:47.382432 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.382432    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9dc4580-c625-4505-950d-e7579d3d2af9-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:47.382696 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.382447    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdf2v\" (UniqueName: \"kubernetes.io/projected/a9dc4580-c625-4505-950d-e7579d3d2af9-kube-api-access-sdf2v\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:47.410499 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:47.410473    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"]
Oct 30 12:55:47.413650 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:55:47.413621    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43afe350_83fe_4157_9ebc_1f36b9a9d008.slice/crio-7fe6e45f8fbf9850ca1951e8aaf80b440bc3bb55a264798877ecf7fd817d3cd2 WatchSource:0}: Error finding container 7fe6e45f8fbf9850ca1951e8aaf80b440bc3bb55a264798877ecf7fd817d3cd2: Status 404 returned error can't find the container with id 7fe6e45f8fbf9850ca1951e8aaf80b440bc3bb55a264798877ecf7fd817d3cd2
Oct 30 12:55:48.015748 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.015703    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"]
Oct 30 12:55:48.016291 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.016271    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a9dc4580-c625-4505-950d-e7579d3d2af9" containerName="revoke-keys"
Oct 30 12:55:48.016291 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.016291    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9dc4580-c625-4505-950d-e7579d3d2af9" containerName="revoke-keys"
Oct 30 12:55:48.016464 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.016409    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9dc4580-c625-4505-950d-e7579d3d2af9" containerName="revoke-keys"
Oct 30 12:55:48.025171 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.025135    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.031376 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.031342    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"
Oct 30 12:55:48.031519 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.031344    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl" event={"ID":"a9dc4580-c625-4505-950d-e7579d3d2af9","Type":"ContainerDied","Data":"440f48f49f80870b6cd267f9464340d1a113b44a038bd96c25bc5a39af1d9435"}
Oct 30 12:55:48.031519 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.031482    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="440f48f49f80870b6cd267f9464340d1a113b44a038bd96c25bc5a39af1d9435"
Oct 30 12:55:48.033364 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.033330    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc" event={"ID":"43afe350-83fe-4157-9ebc-1f36b9a9d008","Type":"ContainerStarted","Data":"a4d3abc93a27db8296664ecd27d09917fb63beccc27490c570289aeb466ee26d"}
Oct 30 12:55:48.033517 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.033386    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc" event={"ID":"43afe350-83fe-4157-9ebc-1f36b9a9d008","Type":"ContainerStarted","Data":"7fe6e45f8fbf9850ca1951e8aaf80b440bc3bb55a264798877ecf7fd817d3cd2"}
Oct 30 12:55:48.042452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.042421    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"]
Oct 30 12:55:48.093336 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.093301    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6a311e-4140-468f-a44f-311e27151550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.093824 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.093702    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a311e-4140-468f-a44f-311e27151550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.093873 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.093827    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnh4\" (UniqueName: \"kubernetes.io/projected/5b6a311e-4140-468f-a44f-311e27151550-kube-api-access-blnh4\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.141507 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.141450    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc" podStartSLOduration=2.141432076 podStartE2EDuration="2.141432076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:48.140798962 +0000 UTC m=+1412.616518579" watchObservedRunningTime="2026-10-30 12:55:48.141432076 +0000 UTC m=+1412.617151685"
Oct 30 12:55:48.194612 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.194508    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6a311e-4140-468f-a44f-311e27151550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.194779 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.194624    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a311e-4140-468f-a44f-311e27151550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.194779 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.194659    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnh4\" (UniqueName: \"kubernetes.io/projected/5b6a311e-4140-468f-a44f-311e27151550-kube-api-access-blnh4\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.195410 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.195383    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a311e-4140-468f-a44f-311e27151550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.197173 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.197148    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6a311e-4140-468f-a44f-311e27151550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.220366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.220333    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnh4\" (UniqueName: \"kubernetes.io/projected/5b6a311e-4140-468f-a44f-311e27151550-kube-api-access-blnh4\") pod \"maas-api-revoke-sub-b4ee3e44-7jk5c\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.341969 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.341932    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:48.656257 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:48.656227    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"]
Oct 30 12:55:48.658396 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:55:48.658366    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6a311e_4140_468f_a44f_311e27151550.slice/crio-f870a91877b44509381660e398f5f2c5f39428843b42a87036d840232d8c392f WatchSource:0}: Error finding container f870a91877b44509381660e398f5f2c5f39428843b42a87036d840232d8c392f: Status 404 returned error can't find the container with id f870a91877b44509381660e398f5f2c5f39428843b42a87036d840232d8c392f
Oct 30 12:55:49.041171 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:49.041133    4993 generic.go:356] "Generic (PLEG): container finished" podID="43afe350-83fe-4157-9ebc-1f36b9a9d008" containerID="a4d3abc93a27db8296664ecd27d09917fb63beccc27490c570289aeb466ee26d" exitCode=0
Oct 30 12:55:49.041387 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:49.041227    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc" event={"ID":"43afe350-83fe-4157-9ebc-1f36b9a9d008","Type":"ContainerDied","Data":"a4d3abc93a27db8296664ecd27d09917fb63beccc27490c570289aeb466ee26d"}
Oct 30 12:55:49.042938 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:49.042914    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c" event={"ID":"5b6a311e-4140-468f-a44f-311e27151550","Type":"ContainerStarted","Data":"a1ba9b86667366a6fdbd42108f799efc3c22e90c650b900eab0abdab932ef894"}
Oct 30 12:55:49.043034 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:49.042957    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c" event={"ID":"5b6a311e-4140-468f-a44f-311e27151550","Type":"ContainerStarted","Data":"f870a91877b44509381660e398f5f2c5f39428843b42a87036d840232d8c392f"}
Oct 30 12:55:49.131887 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:49.131833    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c" podStartSLOduration=2.131818113 podStartE2EDuration="2.131818113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:49.13085694 +0000 UTC m=+1413.606576551" watchObservedRunningTime="2026-10-30 12:55:49.131818113 +0000 UTC m=+1413.607537722"
Oct 30 12:55:50.056042 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.055999    4993 generic.go:356] "Generic (PLEG): container finished" podID="5b6a311e-4140-468f-a44f-311e27151550" containerID="a1ba9b86667366a6fdbd42108f799efc3c22e90c650b900eab0abdab932ef894" exitCode=0
Oct 30 12:55:50.059482 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.056498    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c" event={"ID":"5b6a311e-4140-468f-a44f-311e27151550","Type":"ContainerDied","Data":"a1ba9b86667366a6fdbd42108f799efc3c22e90c650b900eab0abdab932ef894"}
Oct 30 12:55:50.288932 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.288880    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:50.421381 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.420690    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-openshift-ca\") pod \"43afe350-83fe-4157-9ebc-1f36b9a9d008\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") "
Oct 30 12:55:50.421381 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.420794    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43afe350-83fe-4157-9ebc-1f36b9a9d008-kube-api-access-9x8f2\" (UniqueName: \"kubernetes.io/projected/43afe350-83fe-4157-9ebc-1f36b9a9d008-kube-api-access-9x8f2\") pod \"43afe350-83fe-4157-9ebc-1f36b9a9d008\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") "
Oct 30 12:55:50.421381 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.420984    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-xks-ca\") pod \"43afe350-83fe-4157-9ebc-1f36b9a9d008\" (UID: \"43afe350-83fe-4157-9ebc-1f36b9a9d008\") "
Oct 30 12:55:50.422328 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.422284    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-openshift-ca" pod "43afe350-83fe-4157-9ebc-1f36b9a9d008" (UID: "43afe350-83fe-4157-9ebc-1f36b9a9d008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:50.426693 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.426664    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43afe350-83fe-4157-9ebc-1f36b9a9d008-kube-api-access-9x8f2" pod "43afe350-83fe-4157-9ebc-1f36b9a9d008" (UID: "43afe350-83fe-4157-9ebc-1f36b9a9d008"). InnerVolumeSpecName "kube-api-access-9x8f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:50.426860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.426688    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-xks-ca" pod "43afe350-83fe-4157-9ebc-1f36b9a9d008" (UID: "43afe350-83fe-4157-9ebc-1f36b9a9d008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:50.522513 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.522468    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:50.522513 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.522519    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8f2\" (UniqueName: \"kubernetes.io/projected/43afe350-83fe-4157-9ebc-1f36b9a9d008-kube-api-access-9x8f2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:50.522781 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:50.522533    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afe350-83fe-4157-9ebc-1f36b9a9d008-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:51.068051 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.068020    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"
Oct 30 12:55:51.068249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.068016    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc" event={"ID":"43afe350-83fe-4157-9ebc-1f36b9a9d008","Type":"ContainerDied","Data":"7fe6e45f8fbf9850ca1951e8aaf80b440bc3bb55a264798877ecf7fd817d3cd2"}
Oct 30 12:55:51.068249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.068158    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe6e45f8fbf9850ca1951e8aaf80b440bc3bb55a264798877ecf7fd817d3cd2"
Oct 30 12:55:51.225395 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.225367    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:51.332408 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.332296    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6a311e-4140-468f-a44f-311e27151550-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a311e-4140-468f-a44f-311e27151550-maas-api-openshift-ca\") pod \"5b6a311e-4140-468f-a44f-311e27151550\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") "
Oct 30 12:55:51.332408 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.332376    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6a311e-4140-468f-a44f-311e27151550-kube-api-access-blnh4\" (UniqueName: \"kubernetes.io/projected/5b6a311e-4140-468f-a44f-311e27151550-kube-api-access-blnh4\") pod \"5b6a311e-4140-468f-a44f-311e27151550\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") "
Oct 30 12:55:51.332938 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.332451    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6a311e-4140-468f-a44f-311e27151550-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6a311e-4140-468f-a44f-311e27151550-maas-api-xks-ca\") pod \"5b6a311e-4140-468f-a44f-311e27151550\" (UID: \"5b6a311e-4140-468f-a44f-311e27151550\") "
Oct 30 12:55:51.332938 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.332772    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6a311e-4140-468f-a44f-311e27151550-maas-api-openshift-ca" pod "5b6a311e-4140-468f-a44f-311e27151550" (UID: "5b6a311e-4140-468f-a44f-311e27151550"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:51.334601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.334576    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6a311e-4140-468f-a44f-311e27151550-maas-api-xks-ca" pod "5b6a311e-4140-468f-a44f-311e27151550" (UID: "5b6a311e-4140-468f-a44f-311e27151550"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:51.334711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.334695    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6a311e-4140-468f-a44f-311e27151550-kube-api-access-blnh4" pod "5b6a311e-4140-468f-a44f-311e27151550" (UID: "5b6a311e-4140-468f-a44f-311e27151550"). InnerVolumeSpecName "kube-api-access-blnh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:51.433374 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.433331    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6a311e-4140-468f-a44f-311e27151550-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:51.433538 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.433388    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a311e-4140-468f-a44f-311e27151550-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:51.433538 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:51.433399    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blnh4\" (UniqueName: \"kubernetes.io/projected/5b6a311e-4140-468f-a44f-311e27151550-kube-api-access-blnh4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:52.076392 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:52.076360    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"
Oct 30 12:55:52.076581 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:52.076391    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c" event={"ID":"5b6a311e-4140-468f-a44f-311e27151550","Type":"ContainerDied","Data":"f870a91877b44509381660e398f5f2c5f39428843b42a87036d840232d8c392f"}
Oct 30 12:55:52.076581 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:52.076451    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f870a91877b44509381660e398f5f2c5f39428843b42a87036d840232d8c392f"
Oct 30 12:55:56.275330 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.275280    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"]
Oct 30 12:55:56.275862 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.275844    4993 cpu_manager.go:414] "containerMap: removing container" podUID="43afe350-83fe-4157-9ebc-1f36b9a9d008" containerName="revoke-keys"
Oct 30 12:55:56.275862 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.275859    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43afe350-83fe-4157-9ebc-1f36b9a9d008" containerName="revoke-keys"
Oct 30 12:55:56.275936 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.275877    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5b6a311e-4140-468f-a44f-311e27151550" containerName="revoke-keys"
Oct 30 12:55:56.275936 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.275883    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6a311e-4140-468f-a44f-311e27151550" containerName="revoke-keys"
Oct 30 12:55:56.276017 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.275963    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6a311e-4140-468f-a44f-311e27151550" containerName="revoke-keys"
Oct 30 12:55:56.276017 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.275978    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="43afe350-83fe-4157-9ebc-1f36b9a9d008" containerName="revoke-keys"
Oct 30 12:55:56.279571 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.279537    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.304599 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.304553    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"]
Oct 30 12:55:56.313346 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.313310    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:55:56.313546 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.313399    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:55:56.313546 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.313523    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:55:56.313666 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.313615    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:55:56.389055 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.389011    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.389272 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.389080    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.389272 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.389163    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljx5c\" (UniqueName: \"kubernetes.io/projected/3da156ee-bcee-44c3-821c-37b2870f8830-kube-api-access-ljx5c\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.490822 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.490750    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.491094 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.490871    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.491094 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.490918    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljx5c\" (UniqueName: \"kubernetes.io/projected/3da156ee-bcee-44c3-821c-37b2870f8830-kube-api-access-ljx5c\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.491656 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.491627    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.493921 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.493882    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.517492 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.517461    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljx5c\" (UniqueName: \"kubernetes.io/projected/3da156ee-bcee-44c3-821c-37b2870f8830-kube-api-access-ljx5c\") pod \"maas-api-revoke-sub-f6b51790-pfxwr\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.594097 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.593996    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:56.910407 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:56.910381    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"]
Oct 30 12:55:56.912014 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:55:56.911979    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da156ee_bcee_44c3_821c_37b2870f8830.slice/crio-3d3230e255b2018caf5e58afc0e4ef662c9ca3ade534d54c01ab3d5358260f40 WatchSource:0}: Error finding container 3d3230e255b2018caf5e58afc0e4ef662c9ca3ade534d54c01ab3d5358260f40: Status 404 returned error can't find the container with id 3d3230e255b2018caf5e58afc0e4ef662c9ca3ade534d54c01ab3d5358260f40
Oct 30 12:55:57.108794 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:57.108757    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr" event={"ID":"3da156ee-bcee-44c3-821c-37b2870f8830","Type":"ContainerStarted","Data":"3d3230e255b2018caf5e58afc0e4ef662c9ca3ade534d54c01ab3d5358260f40"}
Oct 30 12:55:58.116146 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.116111    4993 generic.go:356] "Generic (PLEG): container finished" podID="3da156ee-bcee-44c3-821c-37b2870f8830" containerID="1bcf142e8bed2460ccc50a534653e91af1f09bb531a8fcd78a47b5350f749dcf" exitCode=0
Oct 30 12:55:58.116576 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.116198    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr" event={"ID":"3da156ee-bcee-44c3-821c-37b2870f8830","Type":"ContainerDied","Data":"1bcf142e8bed2460ccc50a534653e91af1f09bb531a8fcd78a47b5350f749dcf"}
Oct 30 12:55:58.374466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.374366    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"]
Oct 30 12:55:58.378065 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.378039    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.399090 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.399058    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"]
Oct 30 12:55:58.512312 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.512267    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.512507 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.512408    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7fb\" (UniqueName: \"kubernetes.io/projected/4e98edd3-7703-4368-8f6c-55d669f33787-kube-api-access-jr7fb\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.512507 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.512483    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.614260 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.614190    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.614456 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.614324    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7fb\" (UniqueName: \"kubernetes.io/projected/4e98edd3-7703-4368-8f6c-55d669f33787-kube-api-access-jr7fb\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.614456 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.614424    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.614941 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.614920    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.616854 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.616833    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.639780 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.639716    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7fb\" (UniqueName: \"kubernetes.io/projected/4e98edd3-7703-4368-8f6c-55d669f33787-kube-api-access-jr7fb\") pod \"maas-api-revoke-sub-aa667de1-9dl24\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:58.699327 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:58.699289    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:55:59.019278 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.019245    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"]
Oct 30 12:55:59.020668 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:55:59.020634    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e98edd3_7703_4368_8f6c_55d669f33787.slice/crio-a17afad642fad8696a86aac31e16bc97240a1ab5b30797f2a177dc59b7a63ac7 WatchSource:0}: Error finding container a17afad642fad8696a86aac31e16bc97240a1ab5b30797f2a177dc59b7a63ac7: Status 404 returned error can't find the container with id a17afad642fad8696a86aac31e16bc97240a1ab5b30797f2a177dc59b7a63ac7
Oct 30 12:55:59.124403 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.124369    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24" event={"ID":"4e98edd3-7703-4368-8f6c-55d669f33787","Type":"ContainerStarted","Data":"a17afad642fad8696a86aac31e16bc97240a1ab5b30797f2a177dc59b7a63ac7"}
Oct 30 12:55:59.263190 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.263157    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:55:59.322062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.321971    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-xks-ca\") pod \"3da156ee-bcee-44c3-821c-37b2870f8830\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") "
Oct 30 12:55:59.322062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.322018    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da156ee-bcee-44c3-821c-37b2870f8830-kube-api-access-ljx5c\" (UniqueName: \"kubernetes.io/projected/3da156ee-bcee-44c3-821c-37b2870f8830-kube-api-access-ljx5c\") pod \"3da156ee-bcee-44c3-821c-37b2870f8830\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") "
Oct 30 12:55:59.322317 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.322125    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-openshift-ca\") pod \"3da156ee-bcee-44c3-821c-37b2870f8830\" (UID: \"3da156ee-bcee-44c3-821c-37b2870f8830\") "
Oct 30 12:55:59.322502 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.322467    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-openshift-ca" pod "3da156ee-bcee-44c3-821c-37b2870f8830" (UID: "3da156ee-bcee-44c3-821c-37b2870f8830"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:55:59.324318 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.324291    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-xks-ca" pod "3da156ee-bcee-44c3-821c-37b2870f8830" (UID: "3da156ee-bcee-44c3-821c-37b2870f8830"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:59.324405 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.324315    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da156ee-bcee-44c3-821c-37b2870f8830-kube-api-access-ljx5c" pod "3da156ee-bcee-44c3-821c-37b2870f8830" (UID: "3da156ee-bcee-44c3-821c-37b2870f8830"). InnerVolumeSpecName "kube-api-access-ljx5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:59.423812 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.423774    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:59.423812 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.423807    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da156ee-bcee-44c3-821c-37b2870f8830-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:55:59.423812 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:55:59.423817    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljx5c\" (UniqueName: \"kubernetes.io/projected/3da156ee-bcee-44c3-821c-37b2870f8830-kube-api-access-ljx5c\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:56:00.132718 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:00.132674    4993 generic.go:356] "Generic (PLEG): container finished" podID="4e98edd3-7703-4368-8f6c-55d669f33787" containerID="119297ee4d7e3b964807be47b2f4aafd9c3b2f027d84ea6945ba50f452db361d" exitCode=0
Oct 30 12:56:00.133231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:00.132756    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24" event={"ID":"4e98edd3-7703-4368-8f6c-55d669f33787","Type":"ContainerDied","Data":"119297ee4d7e3b964807be47b2f4aafd9c3b2f027d84ea6945ba50f452db361d"}
Oct 30 12:56:00.134562 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:00.134531    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr" event={"ID":"3da156ee-bcee-44c3-821c-37b2870f8830","Type":"ContainerDied","Data":"3d3230e255b2018caf5e58afc0e4ef662c9ca3ade534d54c01ab3d5358260f40"}
Oct 30 12:56:00.134692 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:00.134563    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"
Oct 30 12:56:00.134692 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:00.134583    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d3230e255b2018caf5e58afc0e4ef662c9ca3ade534d54c01ab3d5358260f40"
Oct 30 12:56:01.285877 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.285849    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:56:01.345582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.345545    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-openshift-ca\") pod \"4e98edd3-7703-4368-8f6c-55d669f33787\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") "
Oct 30 12:56:01.345758 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.345602    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e98edd3-7703-4368-8f6c-55d669f33787-kube-api-access-jr7fb\" (UniqueName: \"kubernetes.io/projected/4e98edd3-7703-4368-8f6c-55d669f33787-kube-api-access-jr7fb\") pod \"4e98edd3-7703-4368-8f6c-55d669f33787\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") "
Oct 30 12:56:01.345758 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.345681    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-xks-ca\") pod \"4e98edd3-7703-4368-8f6c-55d669f33787\" (UID: \"4e98edd3-7703-4368-8f6c-55d669f33787\") "
Oct 30 12:56:01.346012 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.345976    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-openshift-ca" pod "4e98edd3-7703-4368-8f6c-55d669f33787" (UID: "4e98edd3-7703-4368-8f6c-55d669f33787"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:01.347921 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.347888    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e98edd3-7703-4368-8f6c-55d669f33787-kube-api-access-jr7fb" pod "4e98edd3-7703-4368-8f6c-55d669f33787" (UID: "4e98edd3-7703-4368-8f6c-55d669f33787"). InnerVolumeSpecName "kube-api-access-jr7fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:01.348256 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.348233    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-xks-ca" pod "4e98edd3-7703-4368-8f6c-55d669f33787" (UID: "4e98edd3-7703-4368-8f6c-55d669f33787"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:56:01.447389 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.447287    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:56:01.447389 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.447325    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr7fb\" (UniqueName: \"kubernetes.io/projected/4e98edd3-7703-4368-8f6c-55d669f33787-kube-api-access-jr7fb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:56:01.447389 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:01.447341    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e98edd3-7703-4368-8f6c-55d669f33787-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:56:02.152483 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:02.152451    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"
Oct 30 12:56:02.152706 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:02.152482    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24" event={"ID":"4e98edd3-7703-4368-8f6c-55d669f33787","Type":"ContainerDied","Data":"a17afad642fad8696a86aac31e16bc97240a1ab5b30797f2a177dc59b7a63ac7"}
Oct 30 12:56:02.152706 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:02.152535    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a17afad642fad8696a86aac31e16bc97240a1ab5b30797f2a177dc59b7a63ac7"
Oct 30 12:56:08.843010 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:56:08.842786    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:56:14.328016 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.327964    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"]
Oct 30 12:56:14.328735 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.328709    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4e98edd3-7703-4368-8f6c-55d669f33787" containerName="revoke-keys"
Oct 30 12:56:14.328853 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.328736    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e98edd3-7703-4368-8f6c-55d669f33787" containerName="revoke-keys"
Oct 30 12:56:14.328853 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.328771    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3da156ee-bcee-44c3-821c-37b2870f8830" containerName="revoke-keys"
Oct 30 12:56:14.328853 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.328780    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da156ee-bcee-44c3-821c-37b2870f8830" containerName="revoke-keys"
Oct 30 12:56:14.329020 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.328904    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e98edd3-7703-4368-8f6c-55d669f33787" containerName="revoke-keys"
Oct 30 12:56:14.329020 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.328920    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da156ee-bcee-44c3-821c-37b2870f8830" containerName="revoke-keys"
Oct 30 12:56:14.334325 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.334294    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.372718 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.372682    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"e2e-conflict-a-04ffbbdf-kserve-self-signed-certs\""
Oct 30 12:56:14.377683 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.377649    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"]
Oct 30 12:56:14.446354 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.446306    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-home\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.446522 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.446459    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tls-certs\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.446578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.446556    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvt7\" (UniqueName: \"kubernetes.io/projected/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-kube-api-access-rhvt7\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.446663 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.446645    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tmp-dir\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.446730 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.446715    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-model-cache\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.446770 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.446761    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-dshm\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.548198 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.548160    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-home\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.548420 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.548271    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tls-certs\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.548420 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.548312    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvt7\" (UniqueName: \"kubernetes.io/projected/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-kube-api-access-rhvt7\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.548420 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.548365    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tmp-dir\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.548420 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.548418    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-model-cache\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.548663 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.548459    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-dshm\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.549420 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.549391    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-home\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.550675 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.550365    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-model-cache\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.550831 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.550608    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tmp-dir\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.561277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.553641    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-dshm\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.578610 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.578159    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvt7\" (UniqueName: \"kubernetes.io/projected/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-kube-api-access-rhvt7\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.579862 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.579824    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tls-certs\") pod \"e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.650072 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.650034    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:14.959537 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:14.959505    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"]
Oct 30 12:56:14.961705 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:56:14.961677    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cea23a8_b7a2_45a0_bbe2_e074d03c2bf1.slice/crio-39b29908faac1ba46196558eedebe6fec42c22d92a0fc0d2ce0871a4bd277d6f WatchSource:0}: Error finding container 39b29908faac1ba46196558eedebe6fec42c22d92a0fc0d2ce0871a4bd277d6f: Status 404 returned error can't find the container with id 39b29908faac1ba46196558eedebe6fec42c22d92a0fc0d2ce0871a4bd277d6f
Oct 30 12:56:15.310050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:15.310007    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4" event={"ID":"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1","Type":"ContainerStarted","Data":"7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e"}
Oct 30 12:56:15.310050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:15.310058    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4" event={"ID":"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1","Type":"ContainerStarted","Data":"39b29908faac1ba46196558eedebe6fec42c22d92a0fc0d2ce0871a4bd277d6f"}
Oct 30 12:56:15.358948 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:15.358896    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4" podStartSLOduration=1.358881524 podStartE2EDuration="1.358881524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:15.356914213 +0000 UTC m=+1439.832633823" watchObservedRunningTime="2026-10-30 12:56:15.358881524 +0000 UTC m=+1439.834601134"
Oct 30 12:56:15.650736 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:15.650649    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:24.650277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:24.650232    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:24.662887 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:24.662854    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:25.400264 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:25.400188    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:56:29.012241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:29.011746    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mzhxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:29.012241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:29.011910    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:32.013987 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.013950    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"]
Oct 30 12:56:32.019377 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.019350    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.043669 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.043629    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"distinct-w3-551dd1-kserve-self-signed-certs\""
Oct 30 12:56:32.063290 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.063251    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"]
Oct 30 12:56:32.080146 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.080107    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-home\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.080370 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.080169    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-model-cache\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.080370 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.080298    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-tmp-dir\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.080370 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.080339    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93d6785-85e4-493b-b87f-c73983fde374-tls-certs\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.080370 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.080354    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lzv\" (UniqueName: \"kubernetes.io/projected/d93d6785-85e4-493b-b87f-c73983fde374-kube-api-access-p5lzv\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.080591 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.080372    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-dshm\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.181413 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.181366    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-home\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.181624 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.181431    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-model-cache\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.181624 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.181560    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-tmp-dir\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.181742 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.181621    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93d6785-85e4-493b-b87f-c73983fde374-tls-certs\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.181796 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.181723    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lzv\" (UniqueName: \"kubernetes.io/projected/d93d6785-85e4-493b-b87f-c73983fde374-kube-api-access-p5lzv\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.181850 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.181829    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-home\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.181933 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.181897    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-model-cache\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.182154 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.182104    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-tmp-dir\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.182154 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.182137    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-dshm\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.184256 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.184227    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-dshm\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.184731 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.184707    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93d6785-85e4-493b-b87f-c73983fde374-tls-certs\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.209276 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.209241    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lzv\" (UniqueName: \"kubernetes.io/projected/d93d6785-85e4-493b-b87f-c73983fde374-kube-api-access-p5lzv\") pod \"distinct-w3-551dd1-kserve-565f4c4db-8snn9\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.334287 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.333417    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:32.710141 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:32.710110    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"]
Oct 30 12:56:32.711150 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:56:32.711126    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd93d6785_85e4_493b_b87f_c73983fde374.slice/crio-4475fefa4ef4e10bf600d457cad69a0d35382d10c39dae0e8585ed031f723529 WatchSource:0}: Error finding container 4475fefa4ef4e10bf600d457cad69a0d35382d10c39dae0e8585ed031f723529: Status 404 returned error can't find the container with id 4475fefa4ef4e10bf600d457cad69a0d35382d10c39dae0e8585ed031f723529
Oct 30 12:56:33.479362 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:33.479320    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" event={"ID":"d93d6785-85e4-493b-b87f-c73983fde374","Type":"ContainerStarted","Data":"340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412"}
Oct 30 12:56:33.479362 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:33.479378    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" event={"ID":"d93d6785-85e4-493b-b87f-c73983fde374","Type":"ContainerStarted","Data":"4475fefa4ef4e10bf600d457cad69a0d35382d10c39dae0e8585ed031f723529"}
Oct 30 12:56:33.530379 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:33.530323    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podStartSLOduration=2.530307243 podStartE2EDuration="2.530307243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:33.527563646 +0000 UTC m=+1458.003283256" watchObservedRunningTime="2026-10-30 12:56:33.530307243 +0000 UTC m=+1458.006026852"
Oct 30 12:56:34.334071 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:34.334016    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:36.585231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.578067    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"]
Oct 30 12:56:36.612227 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.611075    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"]
Oct 30 12:56:36.612227 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.611244    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.647356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.645490    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"distinct-w4-e652ca-kserve-self-signed-certs\""
Oct 30 12:56:36.734242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.734178    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tls-certs\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.734532 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.734510    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-dshm\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.734662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.734646    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkg96\" (UniqueName: \"kubernetes.io/projected/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-kube-api-access-pkg96\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.734819 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.734805    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tmp-dir\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.734945 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.734929    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-home\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.735165 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.735147    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-model-cache\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.840902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.840633    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-model-cache\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.840902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.840738    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tls-certs\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.840902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.840771    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-dshm\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.840902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.840793    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkg96\" (UniqueName: \"kubernetes.io/projected/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-kube-api-access-pkg96\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.840902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.840846    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tmp-dir\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.840902 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.840894    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-home\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.842578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.841394    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-home\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.842578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.841635    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-model-cache\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.844241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.842959    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tmp-dir\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.852238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.847887    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tls-certs\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.869486 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.865728    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-dshm\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.869486 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.869391    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkg96\" (UniqueName: \"kubernetes.io/projected/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-kube-api-access-pkg96\") pod \"distinct-w4-e652ca-kserve-56449f47b9-5dmmh\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:36.945524 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:36.945472    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:37.326173 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:37.322390    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"]
Oct 30 12:56:37.327671 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:56:37.327630    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbc2aea1_feb7_4a82_af41_b9173dcf6c71.slice/crio-e24d3604919ea2ec654d8bc834dd7a5c33c5812995da5d22f1c1c8a7fd83e003 WatchSource:0}: Error finding container e24d3604919ea2ec654d8bc834dd7a5c33c5812995da5d22f1c1c8a7fd83e003: Status 404 returned error can't find the container with id e24d3604919ea2ec654d8bc834dd7a5c33c5812995da5d22f1c1c8a7fd83e003
Oct 30 12:56:37.535941 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:37.535466    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" event={"ID":"cbc2aea1-feb7-4a82-af41-b9173dcf6c71","Type":"ContainerStarted","Data":"527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe"}
Oct 30 12:56:37.535941 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:37.535529    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" event={"ID":"cbc2aea1-feb7-4a82-af41-b9173dcf6c71","Type":"ContainerStarted","Data":"e24d3604919ea2ec654d8bc834dd7a5c33c5812995da5d22f1c1c8a7fd83e003"}
Oct 30 12:56:37.586614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:37.584853    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podStartSLOduration=1.584829584 podStartE2EDuration="1.584829584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:37.582472401 +0000 UTC m=+1462.058192012" watchObservedRunningTime="2026-10-30 12:56:37.584829584 +0000 UTC m=+1462.060549196"
Oct 30 12:56:37.946413 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:37.946310    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:38.661231 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:56:38.660422    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:56:39.517374 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:39.517327    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:56:39.517716 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:39.517684    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:42.334982 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:42.334944    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:42.349806 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:42.349773    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:42.613067 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:42.612965    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:56:46.671120 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.671082    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"]
Oct 30 12:56:46.676000 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.675971    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.697555 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.697515    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"]
Oct 30 12:56:46.706785 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.706757    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"e2e-conflict-b-04ffbbdf-kserve-self-signed-certs\""
Oct 30 12:56:46.810823 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.810779    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-tmp-dir\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.811032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.810836    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-model-cache\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.811032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.810934    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847lg\" (UniqueName: \"kubernetes.io/projected/f297f944-f8d7-4559-88ab-b608ebbfa998-kube-api-access-847lg\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.811032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.810975    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-home\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.811236 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.811108    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-dshm\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.811307 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.811240    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f297f944-f8d7-4559-88ab-b608ebbfa998-tls-certs\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.913233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.912875    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-tmp-dir\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.913233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.912945    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-model-cache\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.913233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.912977    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847lg\" (UniqueName: \"kubernetes.io/projected/f297f944-f8d7-4559-88ab-b608ebbfa998-kube-api-access-847lg\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.913233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.913035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-home\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.913233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.913141    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-dshm\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.913233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.913176    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f297f944-f8d7-4559-88ab-b608ebbfa998-tls-certs\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.917229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.914460    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-tmp-dir\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.917229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.914709    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-model-cache\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.917229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.914959    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-home\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.921272 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.920123    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-dshm\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.925233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.923873    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f297f944-f8d7-4559-88ab-b608ebbfa998-tls-certs\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.948222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.946573    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:46.948222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.946797    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847lg\" (UniqueName: \"kubernetes.io/projected/f297f944-f8d7-4559-88ab-b608ebbfa998-kube-api-access-847lg\") pod \"e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") " pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:46.966249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.966080    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:46.992068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:46.992027    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:47.311185 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:47.311155    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"]
Oct 30 12:56:47.626728 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:47.626616    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2" event={"ID":"f297f944-f8d7-4559-88ab-b608ebbfa998","Type":"ContainerStarted","Data":"dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8"}
Oct 30 12:56:47.626728 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:47.626665    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2" event={"ID":"f297f944-f8d7-4559-88ab-b608ebbfa998","Type":"ContainerStarted","Data":"542b014c992f680b507dcfe9833eeea233677c5734a5c5433d1d9c0dfa868e8a"}
Oct 30 12:56:47.638467 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:47.638436    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:56:47.674310 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:47.674230    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2" podStartSLOduration=1.674193013 podStartE2EDuration="1.674193013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:47.672927072 +0000 UTC m=+1472.148646684" watchObservedRunningTime="2026-10-30 12:56:47.674193013 +0000 UTC m=+1472.149912628"
Oct 30 12:56:47.992452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:47.992411    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:56.994229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:56.994156    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:57.019822 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:57.019787    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:56:57.727975 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:56:57.727935    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:57:08.571867 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:57:08.571819    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:57:14.696839 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:14.696798    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"]
Oct 30 12:57:15.868455 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:15.868420    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:57:15.875579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:15.875548    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 12:57:22.277386 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:22.277328    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.389225 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:22.388290    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4" podUID="2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:22.595441 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:22.595312    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.344950 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.344649    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:23.919923 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.919887    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"]
Oct 30 12:57:23.936864 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.936829    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"]
Oct 30 12:57:23.937259 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.937165    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.977865    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.978046    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zb5\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-kube-api-access-t5zb5\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.978078    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.978121    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.978152    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.978177    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.978226    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.978348    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:23.980795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:23.978408    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.000918 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.000877    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7a8b03-openshift-default-dockercfg-mbrk9\""
Oct 30 12:57:24.079554 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.079514    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zb5\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-kube-api-access-t5zb5\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.079810 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.079793    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.079925 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.079911    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.080035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.080019    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.080141 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.080124    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.080272 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.080255    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.080400 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.080385    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.080535 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.080518    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.080640 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.080613    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.080801 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.080783    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.081318 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.081289    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.081438 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.080398    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.081438 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.081330    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.081578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.081558    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.084178 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.084152    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.084805 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.084779    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.107055 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.107021    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zb5\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-kube-api-access-t5zb5\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.109121 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.109063    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.284887 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.284847    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:24.396298 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.378160    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"]
Oct 30 12:57:24.667275 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.667238    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"]
Oct 30 12:57:24.668364 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:57:24.668330    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d46308_8d87_4f1f_b898_0d1dfbba8f03.slice/crio-62e87be22807352241d5487254ef77fbe23666b464442fcc6a17d8891ce19032 WatchSource:0}: Error finding container 62e87be22807352241d5487254ef77fbe23666b464442fcc6a17d8891ce19032: Status 404 returned error can't find the container with id 62e87be22807352241d5487254ef77fbe23666b464442fcc6a17d8891ce19032
Oct 30 12:57:24.670838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.670793    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:24.670952 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.670911    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:24.671018 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:24.670955    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:25.055291 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:25.055246    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" event={"ID":"f9d46308-8d87-4f1f-b898-0d1dfbba8f03","Type":"ContainerStarted","Data":"6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2"}
Oct 30 12:57:25.055539 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:25.055317    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" event={"ID":"f9d46308-8d87-4f1f-b898-0d1dfbba8f03","Type":"ContainerStarted","Data":"62e87be22807352241d5487254ef77fbe23666b464442fcc6a17d8891ce19032"}
Oct 30 12:57:25.286106 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:25.286063    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:25.287938 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:25.287902    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:57:25.288089 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:25.287965    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 12:57:26.286249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:26.285420    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:57:26.286249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:26.285478    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 12:57:27.289267 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.288934    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:57:27.289267 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.289011    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 12:57:27.603911 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.603809    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podStartSLOduration=4.6037923240000005 podStartE2EDuration="4.603792324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:25.140597246 +0000 UTC m=+1509.616316855" watchObservedRunningTime="2026-10-30 12:57:27.603792324 +0000 UTC m=+1512.079511935"
Oct 30 12:57:27.605348 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.605319    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"]
Oct 30 12:57:27.609661 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.609633    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.630097 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.630064    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"]
Oct 30 12:57:27.640356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.640326    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:57:27.640573 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.640348    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:57:27.640642 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.640349    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:57:27.640642 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.640371    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:57:27.680152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.680098    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.680152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.680158    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.680482 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.680235    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkplm\" (UniqueName: \"kubernetes.io/projected/f5546197-1610-45c0-9227-dcdbc6d2c2bc-kube-api-access-pkplm\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.782086 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.782037    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.782086 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.782086    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.782366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.782130    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkplm\" (UniqueName: \"kubernetes.io/projected/f5546197-1610-45c0-9227-dcdbc6d2c2bc-kube-api-access-pkplm\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.783777 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.783747    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.785397 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.785369    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.808675 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.808643    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkplm\" (UniqueName: \"kubernetes.io/projected/f5546197-1610-45c0-9227-dcdbc6d2c2bc-kube-api-access-pkplm\") pod \"maas-api-revoke-sub-318ba379-pkhzb\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:27.924906 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:27.924813    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:28.286137 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:28.285731    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"]
Oct 30 12:57:28.286995 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:28.286964    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:57:28.287238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:28.287159    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 12:57:28.290126 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:57:28.290107    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5546197_1610_45c0_9227_dcdbc6d2c2bc.slice/crio-53764e46eaecf0cfb5ba25b1cc106f24288f0b3fd9eb7053d4e7a7ed89ad66ae WatchSource:0}: Error finding container 53764e46eaecf0cfb5ba25b1cc106f24288f0b3fd9eb7053d4e7a7ed89ad66ae: Status 404 returned error can't find the container with id 53764e46eaecf0cfb5ba25b1cc106f24288f0b3fd9eb7053d4e7a7ed89ad66ae
Oct 30 12:57:29.117425 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:29.117376    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb" event={"ID":"f5546197-1610-45c0-9227-dcdbc6d2c2bc","Type":"ContainerStarted","Data":"5591e5cfc20b3ec29831527367822afbd5099e11b7c8a2808b6be028c2ee74f5"}
Oct 30 12:57:29.117666 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:29.117438    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb" event={"ID":"f5546197-1610-45c0-9227-dcdbc6d2c2bc","Type":"ContainerStarted","Data":"53764e46eaecf0cfb5ba25b1cc106f24288f0b3fd9eb7053d4e7a7ed89ad66ae"}
Oct 30 12:57:29.285316 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:29.285273    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:57:29.285494 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:29.285336    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 12:57:29.723959 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:29.723864    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2" podUID="f297f944-f8d7-4559-88ab-b608ebbfa998" containerName="main" containerID="cri-o://dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8" gracePeriod=45
Oct 30 12:57:30.040071 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.040035    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:57:30.092115 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.092048    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb" podStartSLOduration=3.09202402 podStartE2EDuration="3.09202402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:29.170502449 +0000 UTC m=+1513.646222063" watchObservedRunningTime="2026-10-30 12:57:30.09202402 +0000 UTC m=+1514.567743631"
Oct 30 12:57:30.110296 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110248    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-tmp-dir\") pod \"f297f944-f8d7-4559-88ab-b608ebbfa998\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") "
Oct 30 12:57:30.110515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110324    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f297f944-f8d7-4559-88ab-b608ebbfa998-kube-api-access-847lg\" (UniqueName: \"kubernetes.io/projected/f297f944-f8d7-4559-88ab-b608ebbfa998-kube-api-access-847lg\") pod \"f297f944-f8d7-4559-88ab-b608ebbfa998\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") "
Oct 30 12:57:30.110515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110366    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-model-cache\") pod \"f297f944-f8d7-4559-88ab-b608ebbfa998\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") "
Oct 30 12:57:30.110515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110452    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-dshm\") pod \"f297f944-f8d7-4559-88ab-b608ebbfa998\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") "
Oct 30 12:57:30.110515 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110482    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-home\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-home\") pod \"f297f944-f8d7-4559-88ab-b608ebbfa998\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") "
Oct 30 12:57:30.110822 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110595    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f297f944-f8d7-4559-88ab-b608ebbfa998-tls-certs\" (UniqueName: \"kubernetes.io/secret/f297f944-f8d7-4559-88ab-b608ebbfa998-tls-certs\") pod \"f297f944-f8d7-4559-88ab-b608ebbfa998\" (UID: \"f297f944-f8d7-4559-88ab-b608ebbfa998\") "
Oct 30 12:57:30.110822 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110601    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-tmp-dir" pod "f297f944-f8d7-4559-88ab-b608ebbfa998" (UID: "f297f944-f8d7-4559-88ab-b608ebbfa998"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:30.110932 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110815    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-model-cache" pod "f297f944-f8d7-4559-88ab-b608ebbfa998" (UID: "f297f944-f8d7-4559-88ab-b608ebbfa998"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:30.110932 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110908    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:30.110932 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.110926    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:30.111411 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.111385    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-home" pod "f297f944-f8d7-4559-88ab-b608ebbfa998" (UID: "f297f944-f8d7-4559-88ab-b608ebbfa998"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:30.113962 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.113916    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f297f944-f8d7-4559-88ab-b608ebbfa998-tls-certs" pod "f297f944-f8d7-4559-88ab-b608ebbfa998" (UID: "f297f944-f8d7-4559-88ab-b608ebbfa998"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:30.114680 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.114187    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f297f944-f8d7-4559-88ab-b608ebbfa998-kube-api-access-847lg" pod "f297f944-f8d7-4559-88ab-b608ebbfa998" (UID: "f297f944-f8d7-4559-88ab-b608ebbfa998"). InnerVolumeSpecName "kube-api-access-847lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:30.114680 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.114573    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-dshm" pod "f297f944-f8d7-4559-88ab-b608ebbfa998" (UID: "f297f944-f8d7-4559-88ab-b608ebbfa998"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:30.127861 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.127827    4993 generic.go:356] "Generic (PLEG): container finished" podID="f5546197-1610-45c0-9227-dcdbc6d2c2bc" containerID="5591e5cfc20b3ec29831527367822afbd5099e11b7c8a2808b6be028c2ee74f5" exitCode=0
Oct 30 12:57:30.128018 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.127933    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb" event={"ID":"f5546197-1610-45c0-9227-dcdbc6d2c2bc","Type":"ContainerDied","Data":"5591e5cfc20b3ec29831527367822afbd5099e11b7c8a2808b6be028c2ee74f5"}
Oct 30 12:57:30.129769 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.129739    4993 generic.go:356] "Generic (PLEG): container finished" podID="f297f944-f8d7-4559-88ab-b608ebbfa998" containerID="dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8" exitCode=0
Oct 30 12:57:30.129907 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.129796    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2" event={"ID":"f297f944-f8d7-4559-88ab-b608ebbfa998","Type":"ContainerDied","Data":"dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8"}
Oct 30 12:57:30.129907 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.129821    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2" event={"ID":"f297f944-f8d7-4559-88ab-b608ebbfa998","Type":"ContainerDied","Data":"542b014c992f680b507dcfe9833eeea233677c5734a5c5433d1d9c0dfa868e8a"}
Oct 30 12:57:30.129907 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.129838    4993 scope.go:122] "RemoveContainer" containerID="dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8"
Oct 30 12:57:30.130075 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.129958    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"
Oct 30 12:57:30.191048 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.191012    4993 scope.go:122] "RemoveContainer" containerID="dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8"
Oct 30 12:57:30.191486 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:57:30.191452    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8\": container with ID starting with dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8 not found: ID does not exist" containerID="dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8"
Oct 30 12:57:30.191583 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.191499    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8"} err="failed to get container status \"dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8\": rpc error: code = NotFound desc = could not find container \"dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8\": container with ID starting with dc126cb6b5b7dd438c1cf6c5ab643fe02e5ac460491bc8bb26cd99650db3f4d8 not found: ID does not exist"
Oct 30 12:57:30.211895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.211845    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f297f944-f8d7-4559-88ab-b608ebbfa998-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:30.211895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.211888    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847lg\" (UniqueName: \"kubernetes.io/projected/f297f944-f8d7-4559-88ab-b608ebbfa998-kube-api-access-847lg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:30.211895 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.211902    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:30.212146 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.211914    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f297f944-f8d7-4559-88ab-b608ebbfa998-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:30.240731 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.240694    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"]
Oct 30 12:57:30.248386 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.248354    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-b-04ffbbdf-kserve-6c9f59d475-b6mg2"]
Oct 30 12:57:30.286265 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.286150    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:57:30.286265 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:30.286224    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 12:57:31.286263 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.286193    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 12:57:31.286754 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.286292    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 12:57:31.324919 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.324891    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:31.446329 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.446223    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-openshift-ca\") pod \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") "
Oct 30 12:57:31.446527 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.446377    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-xks-ca\") pod \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") "
Oct 30 12:57:31.446527 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.446458    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5546197-1610-45c0-9227-dcdbc6d2c2bc-kube-api-access-pkplm\" (UniqueName: \"kubernetes.io/projected/f5546197-1610-45c0-9227-dcdbc6d2c2bc-kube-api-access-pkplm\") pod \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\" (UID: \"f5546197-1610-45c0-9227-dcdbc6d2c2bc\") "
Oct 30 12:57:31.446638 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.446597    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-openshift-ca" pod "f5546197-1610-45c0-9227-dcdbc6d2c2bc" (UID: "f5546197-1610-45c0-9227-dcdbc6d2c2bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:31.446817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.446800    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:31.448773 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.448737    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-xks-ca" pod "f5546197-1610-45c0-9227-dcdbc6d2c2bc" (UID: "f5546197-1610-45c0-9227-dcdbc6d2c2bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:31.448939 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.448852    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5546197-1610-45c0-9227-dcdbc6d2c2bc-kube-api-access-pkplm" pod "f5546197-1610-45c0-9227-dcdbc6d2c2bc" (UID: "f5546197-1610-45c0-9227-dcdbc6d2c2bc"). InnerVolumeSpecName "kube-api-access-pkplm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:31.547948 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.547900    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5546197-1610-45c0-9227-dcdbc6d2c2bc-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:31.547948 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.547938    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkplm\" (UniqueName: \"kubernetes.io/projected/f5546197-1610-45c0-9227-dcdbc6d2c2bc-kube-api-access-pkplm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:31.696274 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:31.696237    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f297f944-f8d7-4559-88ab-b608ebbfa998" path="/var/lib/kubelet/pods/f297f944-f8d7-4559-88ab-b608ebbfa998/volumes"
Oct 30 12:57:32.149901 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:32.149858    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb" event={"ID":"f5546197-1610-45c0-9227-dcdbc6d2c2bc","Type":"ContainerDied","Data":"53764e46eaecf0cfb5ba25b1cc106f24288f0b3fd9eb7053d4e7a7ed89ad66ae"}
Oct 30 12:57:32.150112 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:32.149919    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53764e46eaecf0cfb5ba25b1cc106f24288f0b3fd9eb7053d4e7a7ed89ad66ae"
Oct 30 12:57:32.150112 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:32.149980    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"
Oct 30 12:57:32.290904 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:32.290873    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:32.291355 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:32.291332    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:32.292084 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:32.292068    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 12:57:37.305822 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.305775    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"]
Oct 30 12:57:37.306605 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.306581    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f297f944-f8d7-4559-88ab-b608ebbfa998" containerName="main"
Oct 30 12:57:37.306671 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.306610    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f297f944-f8d7-4559-88ab-b608ebbfa998" containerName="main"
Oct 30 12:57:37.306671 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.306642    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f5546197-1610-45c0-9227-dcdbc6d2c2bc" containerName="revoke-keys"
Oct 30 12:57:37.306671 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.306652    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5546197-1610-45c0-9227-dcdbc6d2c2bc" containerName="revoke-keys"
Oct 30 12:57:37.306809 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.306796    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f297f944-f8d7-4559-88ab-b608ebbfa998" containerName="main"
Oct 30 12:57:37.306846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.306817    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5546197-1610-45c0-9227-dcdbc6d2c2bc" containerName="revoke-keys"
Oct 30 12:57:37.310934 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.310903    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.360171 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.360133    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"]
Oct 30 12:57:37.426150 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426099    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.426378 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426257    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.426378 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426298    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.426472 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426414    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba323d-a154-4386-a080-1999960e130c-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.426508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426462    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtrt\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-kube-api-access-9mtrt\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.426580 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426564    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.426618 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426604    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.426666 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426653    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba323d-a154-4386-a080-1999960e130c-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.426713 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.426676    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528404 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528364    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba323d-a154-4386-a080-1999960e130c-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528404 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528403    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtrt\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-kube-api-access-9mtrt\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528452    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528479    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528505    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba323d-a154-4386-a080-1999960e130c-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528533    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528659 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528593    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528919 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528729    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.528919 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528762    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.529021 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.528963    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.529070 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.529050    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.529196 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.529169    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.529349 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.529331    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.529592 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.529566    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba323d-a154-4386-a080-1999960e130c-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.531071 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.531041    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.531253 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.531232    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba323d-a154-4386-a080-1999960e130c-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.565988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.565911    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.566749 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.566722    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtrt\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-kube-api-access-9mtrt\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.626549 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.626504    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:37.963119 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.963085    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"]
Oct 30 12:57:37.963845 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:57:37.963814    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ba323d_a154_4386_a080_1999960e130c.slice/crio-f3e21f8e6c0b363b426f4ce9e37647c67be7a91276f6c9f32c8d18908d93d058 WatchSource:0}: Error finding container f3e21f8e6c0b363b426f4ce9e37647c67be7a91276f6c9f32c8d18908d93d058: Status 404 returned error can't find the container with id f3e21f8e6c0b363b426f4ce9e37647c67be7a91276f6c9f32c8d18908d93d058
Oct 30 12:57:37.966550 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.966513    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:37.966669 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.966605    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:37.966669 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:37.966650    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:38.205994 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:38.205949    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" event={"ID":"07ba323d-a154-4386-a080-1999960e130c","Type":"ContainerStarted","Data":"f1abcef2134913a581188e0daf86c908f15067b01183e84e1cb5ef162a73d234"}
Oct 30 12:57:38.206191 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:38.206012    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" event={"ID":"07ba323d-a154-4386-a080-1999960e130c","Type":"ContainerStarted","Data":"f3e21f8e6c0b363b426f4ce9e37647c67be7a91276f6c9f32c8d18908d93d058"}
Oct 30 12:57:38.256019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:38.255953    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podStartSLOduration=1.255939473 podStartE2EDuration="1.255939473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:38.254967555 +0000 UTC m=+1522.730687167" watchObservedRunningTime="2026-10-30 12:57:38.255939473 +0000 UTC m=+1522.731659156"
Oct 30 12:57:38.597532 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:57:38.597439    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:57:38.627524 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:38.627483    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:39.114398 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.114359    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"]
Oct 30 12:57:39.119129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.119096    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.140007 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.139720    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:57:39.140423 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.140403    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:57:39.144491 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.144437    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"]
Oct 30 12:57:39.158236 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.152324    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:57:39.158236 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.152336    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:57:39.273232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.263950    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.273232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.264057    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6mv\" (UniqueName: \"kubernetes.io/projected/795cf42f-68d3-46bf-ac6e-90ce64f564aa-kube-api-access-gh6mv\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.273232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.264125    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.373290 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.367984    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.373290 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.368114    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6mv\" (UniqueName: \"kubernetes.io/projected/795cf42f-68d3-46bf-ac6e-90ce64f564aa-kube-api-access-gh6mv\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.373290 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.368200    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.373290 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.369682    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.405507 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.405104    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.406831 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.406743    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6mv\" (UniqueName: \"kubernetes.io/projected/795cf42f-68d3-46bf-ac6e-90ce64f564aa-kube-api-access-gh6mv\") pod \"maas-api-revoke-sub-e16fdbbb-4v54z\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.421942 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.421904    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4" podUID="2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" containerName="main" containerID="cri-o://7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e" gracePeriod=45
Oct 30 12:57:39.445243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.445195    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:39.631035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.630886    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:57:39.631035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.630958    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 12:57:39.791781 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.791735    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"]
Oct 30 12:57:39.792740 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:57:39.792705    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795cf42f_68d3_46bf_ac6e_90ce64f564aa.slice/crio-b58d54e663c53ec29993b9eb5b2912a0648652217f2be675f3f3bc81e49f68ec WatchSource:0}: Error finding container b58d54e663c53ec29993b9eb5b2912a0648652217f2be675f3f3bc81e49f68ec: Status 404 returned error can't find the container with id b58d54e663c53ec29993b9eb5b2912a0648652217f2be675f3f3bc81e49f68ec
Oct 30 12:57:39.797304 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.797273    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:57:39.893093 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.892997    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-home\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-home\") pod \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") "
Oct 30 12:57:39.893356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.893340    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-kube-api-access-rhvt7\" (UniqueName: \"kubernetes.io/projected/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-kube-api-access-rhvt7\") pod \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") "
Oct 30 12:57:39.893452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.893300    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-home" pod "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" (UID: "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:39.893789 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.893771    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tls-certs\") pod \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") "
Oct 30 12:57:39.894323 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.894163    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:39.990480 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.982025    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-kube-api-access-rhvt7" pod "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" (UID: "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1"). InnerVolumeSpecName "kube-api-access-rhvt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:39.990480 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.987474    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tls-certs" pod "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" (UID: "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:39.998230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.994662    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-model-cache\") pod \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") "
Oct 30 12:57:39.998230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.994866    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tmp-dir\") pod \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") "
Oct 30 12:57:39.998230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.995094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-dshm\") pod \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\" (UID: \"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1\") "
Oct 30 12:57:40.006234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:39.999970    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-model-cache" pod "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" (UID: "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:40.006234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.000275    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tmp-dir" pod "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" (UID: "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:40.006234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.000711    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:40.006234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.000735    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:40.006234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.000752    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvt7\" (UniqueName: \"kubernetes.io/projected/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-kube-api-access-rhvt7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:40.006234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.000765    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:40.047230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.045655    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-dshm" pod "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" (UID: "2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:57:40.101803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.101767    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:40.225776 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.225744    4993 generic.go:356] "Generic (PLEG): container finished" podID="2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" containerID="7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e" exitCode=0
Oct 30 12:57:40.225985 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.225798    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"
Oct 30 12:57:40.225985 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.225839    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4" event={"ID":"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1","Type":"ContainerDied","Data":"7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e"}
Oct 30 12:57:40.225985 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.225909    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4" event={"ID":"2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1","Type":"ContainerDied","Data":"39b29908faac1ba46196558eedebe6fec42c22d92a0fc0d2ce0871a4bd277d6f"}
Oct 30 12:57:40.225985 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.225935    4993 scope.go:122] "RemoveContainer" containerID="7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e"
Oct 30 12:57:40.228017 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.227977    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z" event={"ID":"795cf42f-68d3-46bf-ac6e-90ce64f564aa","Type":"ContainerStarted","Data":"dcd5abe69e2c17d843ce89e57bbaac62d4436da096af70b546ed8ca75b47d1af"}
Oct 30 12:57:40.228156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.228052    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z" event={"ID":"795cf42f-68d3-46bf-ac6e-90ce64f564aa","Type":"ContainerStarted","Data":"b58d54e663c53ec29993b9eb5b2912a0648652217f2be675f3f3bc81e49f68ec"}
Oct 30 12:57:40.236626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.236240    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"]
Oct 30 12:57:40.236988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.236963    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" containerName="main"
Oct 30 12:57:40.236988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.236988    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" containerName="main"
Oct 30 12:57:40.237321 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.237301    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" containerName="main"
Oct 30 12:57:40.238835 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.238809    4993 scope.go:122] "RemoveContainer" containerID="7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e"
Oct 30 12:57:40.239271 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:57:40.239200    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e\": container with ID starting with 7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e not found: ID does not exist" containerID="7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e"
Oct 30 12:57:40.239337 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.239286    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e"} err="failed to get container status \"7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e\": rpc error: code = NotFound desc = could not find container \"7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e\": container with ID starting with 7fb697c65a5282b39266ae28babbd9820aaab2a9a1d574c0d4c4221c8fd08a7e not found: ID does not exist"
Oct 30 12:57:40.242222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.242190    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.277457 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.277416    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"]
Oct 30 12:57:40.301486 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.301438    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"]
Oct 30 12:57:40.307298 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.307161    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/e2e-conflict-a-04ffbbdf-kserve-f5c6c6fd5-2w8f4"]
Oct 30 12:57:40.386404 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.386306    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z" podStartSLOduration=1.386192817 podStartE2EDuration="1.386192817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:40.375959094 +0000 UTC m=+1524.851678704" watchObservedRunningTime="2026-10-30 12:57:40.386192817 +0000 UTC m=+1524.861912428"
Oct 30 12:57:40.405509 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.405463    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.405883 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.405854    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.405982 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.405906    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnpqh\" (UniqueName: \"kubernetes.io/projected/1c5ea238-7888-48d6-94d0-21c4216e43a5-kube-api-access-bnpqh\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.506749 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.506696    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.506988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.506767    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.506988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.506805    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnpqh\" (UniqueName: \"kubernetes.io/projected/1c5ea238-7888-48d6-94d0-21c4216e43a5-kube-api-access-bnpqh\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.507701 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.507675    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.509691 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.509660    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.532613 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.532570    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnpqh\" (UniqueName: \"kubernetes.io/projected/1c5ea238-7888-48d6-94d0-21c4216e43a5-kube-api-access-bnpqh\") pod \"maas-api-revoke-sub-8cd05489-cqdk5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.576232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.568418    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:40.627077 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.627036    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:57:40.627320 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.627096    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 12:57:40.946405 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:57:40.946364    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5ea238_7888_48d6_94d0_21c4216e43a5.slice/crio-fad1b2b2ea72e98cd64d04fe81297541355d9ec4d2b71533d2975da9276bc3fe WatchSource:0}: Error finding container fad1b2b2ea72e98cd64d04fe81297541355d9ec4d2b71533d2975da9276bc3fe: Status 404 returned error can't find the container with id fad1b2b2ea72e98cd64d04fe81297541355d9ec4d2b71533d2975da9276bc3fe
Oct 30 12:57:40.946947 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:40.946917    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"]
Oct 30 12:57:41.236248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.236185    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5" event={"ID":"1c5ea238-7888-48d6-94d0-21c4216e43a5","Type":"ContainerStarted","Data":"fad1b2b2ea72e98cd64d04fe81297541355d9ec4d2b71533d2975da9276bc3fe"}
Oct 30 12:57:41.239081 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.239052    4993 generic.go:356] "Generic (PLEG): container finished" podID="795cf42f-68d3-46bf-ac6e-90ce64f564aa" containerID="dcd5abe69e2c17d843ce89e57bbaac62d4436da096af70b546ed8ca75b47d1af" exitCode=0
Oct 30 12:57:41.239364 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.239096    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z" event={"ID":"795cf42f-68d3-46bf-ac6e-90ce64f564aa","Type":"ContainerDied","Data":"dcd5abe69e2c17d843ce89e57bbaac62d4436da096af70b546ed8ca75b47d1af"}
Oct 30 12:57:41.417529 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.417495    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"]
Oct 30 12:57:41.422044 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.422018    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.440697 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.440616    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"distinct-w0-7a8b03-kserve-self-signed-certs\""
Oct 30 12:57:41.441744 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.441718    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"]
Oct 30 12:57:41.524180 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.524126    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-home\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.524180 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.524184    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-dshm\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.524447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.524283    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-tmp-dir\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.524447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.524382    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-model-cache\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.524447 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.524431    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpq2\" (UniqueName: \"kubernetes.io/projected/f7b10838-9748-48ad-af2f-71502ed36435-kube-api-access-jlpq2\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.524607 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.524575    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b10838-9748-48ad-af2f-71502ed36435-tls-certs\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.650234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.643462    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b10838-9748-48ad-af2f-71502ed36435-tls-certs\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.650234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.643543    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-home\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.650234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.643574    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-dshm\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.650234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.643642    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-tmp-dir\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.650234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.643711    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-model-cache\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.650234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.643764    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpq2\" (UniqueName: \"kubernetes.io/projected/f7b10838-9748-48ad-af2f-71502ed36435-kube-api-access-jlpq2\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.666181 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.665568    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-home\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.666181 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.665837    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-tmp-dir\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.666181 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.666115    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-model-cache\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.666639 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.666617    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:57:41.666812 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.666786    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 12:57:41.717283 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.714520    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b10838-9748-48ad-af2f-71502ed36435-tls-certs\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.717283 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.716402    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-dshm\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.722280 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.719850    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpq2\" (UniqueName: \"kubernetes.io/projected/f7b10838-9748-48ad-af2f-71502ed36435-kube-api-access-jlpq2\") pod \"distinct-w0-7a8b03-kserve-759c498594-2lxw4\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:41.722280 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.721873    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1" path="/var/lib/kubelet/pods/2cea23a8-b7a2-45a0-bbe2-e074d03c2bf1/volumes"
Oct 30 12:57:41.741191 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:41.741088    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:42.187231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.175340    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"]
Oct 30 12:57:42.187231 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:57:42.186792    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7b10838_9748_48ad_af2f_71502ed36435.slice/crio-22b1eb535df7a99e5897a6da45e729505f4d01c425a025abd96f15c0647b4cd2 WatchSource:0}: Error finding container 22b1eb535df7a99e5897a6da45e729505f4d01c425a025abd96f15c0647b4cd2: Status 404 returned error can't find the container with id 22b1eb535df7a99e5897a6da45e729505f4d01c425a025abd96f15c0647b4cd2
Oct 30 12:57:42.249630 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.249589    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" event={"ID":"f7b10838-9748-48ad-af2f-71502ed36435","Type":"ContainerStarted","Data":"22b1eb535df7a99e5897a6da45e729505f4d01c425a025abd96f15c0647b4cd2"}
Oct 30 12:57:42.251448 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.251409    4993 generic.go:356] "Generic (PLEG): container finished" podID="1c5ea238-7888-48d6-94d0-21c4216e43a5" containerID="83e075c1f38699ae5ba3d83accc4d8072e8cf3c84e2bd4f3fa9b3277355fee2c" exitCode=0
Oct 30 12:57:42.251601 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.251498    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5" event={"ID":"1c5ea238-7888-48d6-94d0-21c4216e43a5","Type":"ContainerDied","Data":"83e075c1f38699ae5ba3d83accc4d8072e8cf3c84e2bd4f3fa9b3277355fee2c"}
Oct 30 12:57:42.507323 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.507293    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:42.591003 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.589854    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-xks-ca\") pod \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") "
Oct 30 12:57:42.591003 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.589944    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795cf42f-68d3-46bf-ac6e-90ce64f564aa-kube-api-access-gh6mv\" (UniqueName: \"kubernetes.io/projected/795cf42f-68d3-46bf-ac6e-90ce64f564aa-kube-api-access-gh6mv\") pod \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") "
Oct 30 12:57:42.591003 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.590142    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-openshift-ca\") pod \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\" (UID: \"795cf42f-68d3-46bf-ac6e-90ce64f564aa\") "
Oct 30 12:57:42.591435 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.591028    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-openshift-ca" pod "795cf42f-68d3-46bf-ac6e-90ce64f564aa" (UID: "795cf42f-68d3-46bf-ac6e-90ce64f564aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:42.603234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.602538    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-xks-ca" pod "795cf42f-68d3-46bf-ac6e-90ce64f564aa" (UID: "795cf42f-68d3-46bf-ac6e-90ce64f564aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:42.606718 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.606648    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795cf42f-68d3-46bf-ac6e-90ce64f564aa-kube-api-access-gh6mv" pod "795cf42f-68d3-46bf-ac6e-90ce64f564aa" (UID: "795cf42f-68d3-46bf-ac6e-90ce64f564aa"). InnerVolumeSpecName "kube-api-access-gh6mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:42.627355 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.627314    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:57:42.627569 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.627375    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 12:57:42.692403 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.692296    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:42.692403 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.692369    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795cf42f-68d3-46bf-ac6e-90ce64f564aa-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:42.692403 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:42.692388    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh6mv\" (UniqueName: \"kubernetes.io/projected/795cf42f-68d3-46bf-ac6e-90ce64f564aa-kube-api-access-gh6mv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:43.282232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.282151    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z" event={"ID":"795cf42f-68d3-46bf-ac6e-90ce64f564aa","Type":"ContainerDied","Data":"b58d54e663c53ec29993b9eb5b2912a0648652217f2be675f3f3bc81e49f68ec"}
Oct 30 12:57:43.282767 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.282249    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58d54e663c53ec29993b9eb5b2912a0648652217f2be675f3f3bc81e49f68ec"
Oct 30 12:57:43.282767 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.282335    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"
Oct 30 12:57:43.287722 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.287434    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" event={"ID":"f7b10838-9748-48ad-af2f-71502ed36435","Type":"ContainerStarted","Data":"86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d"}
Oct 30 12:57:43.336394 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.336336    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podStartSLOduration=2.336321045 podStartE2EDuration="2.336321045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:43.332730576 +0000 UTC m=+1527.808450187" watchObservedRunningTime="2026-10-30 12:57:43.336321045 +0000 UTC m=+1527.812040694"
Oct 30 12:57:43.500492 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.500449    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:43.610311 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.609247    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5ea238-7888-48d6-94d0-21c4216e43a5-kube-api-access-bnpqh\" (UniqueName: \"kubernetes.io/projected/1c5ea238-7888-48d6-94d0-21c4216e43a5-kube-api-access-bnpqh\") pod \"1c5ea238-7888-48d6-94d0-21c4216e43a5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") "
Oct 30 12:57:43.610311 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.609330    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-xks-ca\") pod \"1c5ea238-7888-48d6-94d0-21c4216e43a5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") "
Oct 30 12:57:43.610311 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.609368    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-openshift-ca\") pod \"1c5ea238-7888-48d6-94d0-21c4216e43a5\" (UID: \"1c5ea238-7888-48d6-94d0-21c4216e43a5\") "
Oct 30 12:57:43.610311 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.610276    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-openshift-ca" pod "1c5ea238-7888-48d6-94d0-21c4216e43a5" (UID: "1c5ea238-7888-48d6-94d0-21c4216e43a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:43.625510 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.623495    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-xks-ca" pod "1c5ea238-7888-48d6-94d0-21c4216e43a5" (UID: "1c5ea238-7888-48d6-94d0-21c4216e43a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:43.628725 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.628688    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5ea238-7888-48d6-94d0-21c4216e43a5-kube-api-access-bnpqh" pod "1c5ea238-7888-48d6-94d0-21c4216e43a5" (UID: "1c5ea238-7888-48d6-94d0-21c4216e43a5"). InnerVolumeSpecName "kube-api-access-bnpqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:43.628991 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.628968    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:57:43.629044 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.629019    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 12:57:43.711110 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.711068    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:43.711110 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.711102    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ea238-7888-48d6-94d0-21c4216e43a5-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:43.711361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.711118    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnpqh\" (UniqueName: \"kubernetes.io/projected/1c5ea238-7888-48d6-94d0-21c4216e43a5-kube-api-access-bnpqh\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:43.741298 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:43.741261    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:44.300230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.298385    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"
Oct 30 12:57:44.300230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.298407    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5" event={"ID":"1c5ea238-7888-48d6-94d0-21c4216e43a5","Type":"ContainerDied","Data":"fad1b2b2ea72e98cd64d04fe81297541355d9ec4d2b71533d2975da9276bc3fe"}
Oct 30 12:57:44.300230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.298471    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad1b2b2ea72e98cd64d04fe81297541355d9ec4d2b71533d2975da9276bc3fe"
Oct 30 12:57:44.363633 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.363588    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"]
Oct 30 12:57:44.364375 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.364348    4993 cpu_manager.go:414] "containerMap: removing container" podUID="795cf42f-68d3-46bf-ac6e-90ce64f564aa" containerName="revoke-keys"
Oct 30 12:57:44.364375 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.364376    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795cf42f-68d3-46bf-ac6e-90ce64f564aa" containerName="revoke-keys"
Oct 30 12:57:44.364566 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.364430    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1c5ea238-7888-48d6-94d0-21c4216e43a5" containerName="revoke-keys"
Oct 30 12:57:44.364566 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.364441    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5ea238-7888-48d6-94d0-21c4216e43a5" containerName="revoke-keys"
Oct 30 12:57:44.364672 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.364608    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="795cf42f-68d3-46bf-ac6e-90ce64f564aa" containerName="revoke-keys"
Oct 30 12:57:44.364672 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.364630    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5ea238-7888-48d6-94d0-21c4216e43a5" containerName="revoke-keys"
Oct 30 12:57:44.368800 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.368772    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.388247 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.388193    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"]
Oct 30 12:57:44.390111 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.390075    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:57:44.390326 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.390183    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:57:44.390326 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.390253    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:57:44.396815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.396786    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:57:44.425455 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.425416    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ggzk\" (UniqueName: \"kubernetes.io/projected/bdd506a1-f907-4021-a981-9c5723bc38c9-kube-api-access-8ggzk\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.426145 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.426116    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.426337 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.426307    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.527846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.527797    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ggzk\" (UniqueName: \"kubernetes.io/projected/bdd506a1-f907-4021-a981-9c5723bc38c9-kube-api-access-8ggzk\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.528065 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.527885    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.528065 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.527994    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.528634 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.528610    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.530413 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.530390    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.553626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.553551    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ggzk\" (UniqueName: \"kubernetes.io/projected/bdd506a1-f907-4021-a981-9c5723bc38c9-kube-api-access-8ggzk\") pod \"maas-api-revoke-sub-42219dde-8pddb\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:44.627280 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.627240    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 12:57:44.627474 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.627306    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 12:57:44.694145 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:44.694098    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:45.008979 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:57:45.008942    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd506a1_f907_4021_a981_9c5723bc38c9.slice/crio-333193d765ed6d825fc9f06baed338b486569d423ad00dc2edb0c9d2a237f062 WatchSource:0}: Error finding container 333193d765ed6d825fc9f06baed338b486569d423ad00dc2edb0c9d2a237f062: Status 404 returned error can't find the container with id 333193d765ed6d825fc9f06baed338b486569d423ad00dc2edb0c9d2a237f062
Oct 30 12:57:45.010399 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:45.010364    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"]
Oct 30 12:57:45.310402 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:45.310365    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb" event={"ID":"bdd506a1-f907-4021-a981-9c5723bc38c9","Type":"ContainerStarted","Data":"333193d765ed6d825fc9f06baed338b486569d423ad00dc2edb0c9d2a237f062"}
Oct 30 12:57:45.631951 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:45.631872    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:45.632424 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:45.632403    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:45.633013 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:45.632992    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 12:57:46.372765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:46.372675    4993 generic.go:356] "Generic (PLEG): container finished" podID="bdd506a1-f907-4021-a981-9c5723bc38c9" containerID="88e5daaf1b28fa9d4eb4b45d9beefcd5dd83460794f362bb9b0cd521f5937e34" exitCode=0
Oct 30 12:57:46.373766 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:46.373743    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb" event={"ID":"bdd506a1-f907-4021-a981-9c5723bc38c9","Type":"ContainerDied","Data":"88e5daaf1b28fa9d4eb4b45d9beefcd5dd83460794f362bb9b0cd521f5937e34"}
Oct 30 12:57:47.685705 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.685623    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:47.809450 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.809410    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-xks-ca\") pod \"bdd506a1-f907-4021-a981-9c5723bc38c9\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") "
Oct 30 12:57:47.809696 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.809462    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd506a1-f907-4021-a981-9c5723bc38c9-kube-api-access-8ggzk\" (UniqueName: \"kubernetes.io/projected/bdd506a1-f907-4021-a981-9c5723bc38c9-kube-api-access-8ggzk\") pod \"bdd506a1-f907-4021-a981-9c5723bc38c9\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") "
Oct 30 12:57:47.809696 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.809581    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-openshift-ca\") pod \"bdd506a1-f907-4021-a981-9c5723bc38c9\" (UID: \"bdd506a1-f907-4021-a981-9c5723bc38c9\") "
Oct 30 12:57:47.809950 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.809923    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-openshift-ca" pod "bdd506a1-f907-4021-a981-9c5723bc38c9" (UID: "bdd506a1-f907-4021-a981-9c5723bc38c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:47.810103 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.810080    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:47.812113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.812082    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd506a1-f907-4021-a981-9c5723bc38c9-kube-api-access-8ggzk" pod "bdd506a1-f907-4021-a981-9c5723bc38c9" (UID: "bdd506a1-f907-4021-a981-9c5723bc38c9"). InnerVolumeSpecName "kube-api-access-8ggzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:47.812261 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.812193    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-xks-ca" pod "bdd506a1-f907-4021-a981-9c5723bc38c9" (UID: "bdd506a1-f907-4021-a981-9c5723bc38c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:47.910945 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.910908    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd506a1-f907-4021-a981-9c5723bc38c9-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:47.910945 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:47.910939    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ggzk\" (UniqueName: \"kubernetes.io/projected/bdd506a1-f907-4021-a981-9c5723bc38c9-kube-api-access-8ggzk\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:48.400178 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:48.400136    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb" event={"ID":"bdd506a1-f907-4021-a981-9c5723bc38c9","Type":"ContainerDied","Data":"333193d765ed6d825fc9f06baed338b486569d423ad00dc2edb0c9d2a237f062"}
Oct 30 12:57:48.400178 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:48.400176    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"
Oct 30 12:57:48.400459 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:48.400201    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="333193d765ed6d825fc9f06baed338b486569d423ad00dc2edb0c9d2a237f062"
Oct 30 12:57:49.513062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.513019    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"]
Oct 30 12:57:49.513634 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.513616    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bdd506a1-f907-4021-a981-9c5723bc38c9" containerName="revoke-keys"
Oct 30 12:57:49.513679 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.513635    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd506a1-f907-4021-a981-9c5723bc38c9" containerName="revoke-keys"
Oct 30 12:57:49.513747 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.513738    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd506a1-f907-4021-a981-9c5723bc38c9" containerName="revoke-keys"
Oct 30 12:57:49.517347 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.517325    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.541754 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.541714    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"]
Oct 30 12:57:49.542358 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.542328    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:57:49.550520 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.550483    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:57:49.550705 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.550606    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:57:49.550843 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.550824    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:57:49.632445 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.632412    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.632653 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.632470    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwsv\" (UniqueName: \"kubernetes.io/projected/2d8985b6-144d-4cae-b778-b32ddb1fd79a-kube-api-access-dfwsv\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.632653 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.632605    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.733662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.733625    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.733830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.733723    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.733830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.733773    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwsv\" (UniqueName: \"kubernetes.io/projected/2d8985b6-144d-4cae-b778-b32ddb1fd79a-kube-api-access-dfwsv\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.734586 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.734561    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.736078 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.736056    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.759812 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.759778    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwsv\" (UniqueName: \"kubernetes.io/projected/2d8985b6-144d-4cae-b778-b32ddb1fd79a-kube-api-access-dfwsv\") pod \"maas-api-revoke-sub-124ed98f-2m56r\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:49.831399 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:49.831310    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:50.147495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:50.147452    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"]
Oct 30 12:57:50.415221 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:50.415116    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r" event={"ID":"2d8985b6-144d-4cae-b778-b32ddb1fd79a","Type":"ContainerStarted","Data":"d08f1c40d411f511be27b267f21f20233db6a375ed3c7eea8838041a03be6818"}
Oct 30 12:57:51.225576 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.225534    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"]
Oct 30 12:57:51.229731 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.229709    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.247567 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.247538    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-551dd1-openshift-default-dockercfg-mnncw\""
Oct 30 12:57:51.268273 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.268237    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"]
Oct 30 12:57:51.364521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364406    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.364521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364478    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.364763 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364554    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.364763 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364602    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.364763 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364729    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.364908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364788    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.364908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364829    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8723b3c3-b64e-483f-9efb-c092aa90aa34-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.364908 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364869    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxhr4\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-kube-api-access-rxhr4\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.365052 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.364935    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.424614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.424580    4993 generic.go:356] "Generic (PLEG): container finished" podID="2d8985b6-144d-4cae-b778-b32ddb1fd79a" containerID="9cff8a3aed5c0028a41e66b15b77c40d838cb5c7c5848d0f0dfe83bbfa50285f" exitCode=0
Oct 30 12:57:51.424815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.424667    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r" event={"ID":"2d8985b6-144d-4cae-b778-b32ddb1fd79a","Type":"ContainerDied","Data":"9cff8a3aed5c0028a41e66b15b77c40d838cb5c7c5848d0f0dfe83bbfa50285f"}
Oct 30 12:57:51.465846 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.465808    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.465910    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.465959    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466186 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466068    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466186 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466149    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466311 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466241    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466311 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466288    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466419 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466314    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8723b3c3-b64e-483f-9efb-c092aa90aa34-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466419 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466374    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466419 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466378    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxhr4\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-kube-api-access-rxhr4\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466570 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466429    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.466893 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466865    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.467025 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.466966    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.467370 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.467344    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8723b3c3-b64e-483f-9efb-c092aa90aa34-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.468413 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.468392    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.469166 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.469111    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.508963 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.508930    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.509314 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.509290    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxhr4\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-kube-api-access-rxhr4\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.545162 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.544280    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:51.751680 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.751638    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:51.783746 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.783717    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:51.897407 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.897377    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"]
Oct 30 12:57:51.897561 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:57:51.897517    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8723b3c3_b64e_483f_9efb_c092aa90aa34.slice/crio-6d37e980bc72e0bf28a1ecdcb13a6792f87e3a1bec0976442943df197790bc2f WatchSource:0}: Error finding container 6d37e980bc72e0bf28a1ecdcb13a6792f87e3a1bec0976442943df197790bc2f: Status 404 returned error can't find the container with id 6d37e980bc72e0bf28a1ecdcb13a6792f87e3a1bec0976442943df197790bc2f
Oct 30 12:57:51.900454 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.900414    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:51.900559 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.900510    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:51.900598 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:51.900558    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:52.437107 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.436403    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" event={"ID":"8723b3c3-b64e-483f-9efb-c092aa90aa34","Type":"ContainerStarted","Data":"f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f"}
Oct 30 12:57:52.437107 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.436473    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" event={"ID":"8723b3c3-b64e-483f-9efb-c092aa90aa34","Type":"ContainerStarted","Data":"6d37e980bc72e0bf28a1ecdcb13a6792f87e3a1bec0976442943df197790bc2f"}
Oct 30 12:57:52.451798 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.451765    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:57:52.532237 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.526046    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podStartSLOduration=1.526022662 podStartE2EDuration="1.526022662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:52.505853373 +0000 UTC m=+1536.981572984" watchObservedRunningTime="2026-10-30 12:57:52.526022662 +0000 UTC m=+1537.001742273"
Oct 30 12:57:52.601280 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.599499    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:57:52.601280 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.600342    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:57:52.601280 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.600507    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 12:57:52.714166 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.714133    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:52.922127 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.922079    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-xks-ca\") pod \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") "
Oct 30 12:57:52.922371 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.922202    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8985b6-144d-4cae-b778-b32ddb1fd79a-kube-api-access-dfwsv\" (UniqueName: \"kubernetes.io/projected/2d8985b6-144d-4cae-b778-b32ddb1fd79a-kube-api-access-dfwsv\") pod \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") "
Oct 30 12:57:52.922452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.922371    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-openshift-ca\") pod \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\" (UID: \"2d8985b6-144d-4cae-b778-b32ddb1fd79a\") "
Oct 30 12:57:52.923562 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.923527    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-openshift-ca" pod "2d8985b6-144d-4cae-b778-b32ddb1fd79a" (UID: "2d8985b6-144d-4cae-b778-b32ddb1fd79a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:57:52.925802 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.925771    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8985b6-144d-4cae-b778-b32ddb1fd79a-kube-api-access-dfwsv" pod "2d8985b6-144d-4cae-b778-b32ddb1fd79a" (UID: "2d8985b6-144d-4cae-b778-b32ddb1fd79a"). InnerVolumeSpecName "kube-api-access-dfwsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:52.927721 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:52.927685    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-xks-ca" pod "2d8985b6-144d-4cae-b778-b32ddb1fd79a" (UID: "2d8985b6-144d-4cae-b778-b32ddb1fd79a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:53.024046 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:53.024005    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:53.024046 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:53.024040    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8985b6-144d-4cae-b778-b32ddb1fd79a-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:53.024046 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:53.024051    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwsv\" (UniqueName: \"kubernetes.io/projected/2d8985b6-144d-4cae-b778-b32ddb1fd79a-kube-api-access-dfwsv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:57:53.446180 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:53.446082    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"
Oct 30 12:57:53.446180 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:53.446102    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r" event={"ID":"2d8985b6-144d-4cae-b778-b32ddb1fd79a","Type":"ContainerDied","Data":"d08f1c40d411f511be27b267f21f20233db6a375ed3c7eea8838041a03be6818"}
Oct 30 12:57:53.446746 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:53.446199    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08f1c40d411f511be27b267f21f20233db6a375ed3c7eea8838041a03be6818"
Oct 30 12:57:53.547791 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:53.547749    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:57:53.548098 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:53.548067    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 12:57:54.545054 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:54.545016    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:57:54.545453 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:54.545083    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 12:57:55.544986 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:55.544946    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:57:55.545179 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:55.545004    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 12:57:56.545068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:56.545029    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:57:56.545284 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:56.545084    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 12:57:57.545650 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:57.545610    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:57:57.546062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:57.545669    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 12:57:58.553264 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:58.553228    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:57:58.553905 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:57:58.553857    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 12:58:00.383411 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:00.383356    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:00.591450 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:00.591403    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:00.591722 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:00.591698    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:00.675353 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:00.675254    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:58:00.677198 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:00.677171    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:58:00.689274 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:00.680480    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 12:58:01.186383 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:01.186337    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:01.186721 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:01.186689    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:01.448837 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:01.448722    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:01.450137 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:01.450111    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:58:01.750774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:01.750376    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:01.750774 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:01.750450    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:02.293373 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:02.293317    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:02.453282 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:02.452847    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 12:58:02.487231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:02.483388    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:02.536231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:02.528947    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:02.644163 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:02.644066    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:58:02.665392 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:02.665331    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:02.698359 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:02.698309    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:02.742324 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:02.742259    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:03.355059 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.351970    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:03.749243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.749183    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"]
Oct 30 12:58:03.749992 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.749968    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2d8985b6-144d-4cae-b778-b32ddb1fd79a" containerName="revoke-keys"
Oct 30 12:58:03.750032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.749996    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8985b6-144d-4cae-b778-b32ddb1fd79a" containerName="revoke-keys"
Oct 30 12:58:03.750192 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.750179    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8985b6-144d-4cae-b778-b32ddb1fd79a" containerName="revoke-keys"
Oct 30 12:58:03.755534 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.755466    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:03.774760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.774718    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"]
Oct 30 12:58:03.780590 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.780546    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:03.781024 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.781004    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:03.781283 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.781258    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:03.787177 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.787159    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:58:03.879481 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.879429    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2ht\" (UniqueName: \"kubernetes.io/projected/be38df56-d9bb-40c0-950e-5a9901eb2030-kube-api-access-vm2ht\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:03.879481 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.879482    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:03.879751 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.879565    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:03.981287 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.981246    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2ht\" (UniqueName: \"kubernetes.io/projected/be38df56-d9bb-40c0-950e-5a9901eb2030-kube-api-access-vm2ht\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:03.981287 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.981290    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:03.981550 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.981377    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:03.981989 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.981964    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:03.984181 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:03.984158    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:04.007792 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:04.007721    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2ht\" (UniqueName: \"kubernetes.io/projected/be38df56-d9bb-40c0-950e-5a9901eb2030-kube-api-access-vm2ht\") pod \"maas-api-revoke-sub-3fc46f53-6jbzq\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:04.074711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:04.074664    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:04.392819 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:04.392766    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe38df56_d9bb_40c0_950e_5a9901eb2030.slice/crio-e4e4b1756b386dc88042f0dbc250b798e74e8204150bf7d17010bc205a813a5d WatchSource:0}: Error finding container e4e4b1756b386dc88042f0dbc250b798e74e8204150bf7d17010bc205a813a5d: Status 404 returned error can't find the container with id e4e4b1756b386dc88042f0dbc250b798e74e8204150bf7d17010bc205a813a5d
Oct 30 12:58:04.395526 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:04.395493    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"]
Oct 30 12:58:04.949035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:04.948919    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq" event={"ID":"be38df56-d9bb-40c0-950e-5a9901eb2030","Type":"ContainerStarted","Data":"3a9f6e9a01c2b5bde871fc94c62ced0fa162a7d7e4589484a3632c00e5f872fc"}
Oct 30 12:58:04.949035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:04.948988    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq" event={"ID":"be38df56-d9bb-40c0-950e-5a9901eb2030","Type":"ContainerStarted","Data":"e4e4b1756b386dc88042f0dbc250b798e74e8204150bf7d17010bc205a813a5d"}
Oct 30 12:58:05.004406 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:05.004350    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq" podStartSLOduration=2.004335151 podStartE2EDuration="2.004335151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:05.001799435 +0000 UTC m=+1549.477519046" watchObservedRunningTime="2026-10-30 12:58:05.004335151 +0000 UTC m=+1549.480054760"
Oct 30 12:58:05.959334 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:05.959302    4993 generic.go:356] "Generic (PLEG): container finished" podID="be38df56-d9bb-40c0-950e-5a9901eb2030" containerID="3a9f6e9a01c2b5bde871fc94c62ced0fa162a7d7e4589484a3632c00e5f872fc" exitCode=0
Oct 30 12:58:05.959334 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:05.959340    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq" event={"ID":"be38df56-d9bb-40c0-950e-5a9901eb2030","Type":"ContainerDied","Data":"3a9f6e9a01c2b5bde871fc94c62ced0fa162a7d7e4589484a3632c00e5f872fc"}
Oct 30 12:58:07.131622 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.131594    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:07.321150 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.321110    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-xks-ca\") pod \"be38df56-d9bb-40c0-950e-5a9901eb2030\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") "
Oct 30 12:58:07.321375 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.321186    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be38df56-d9bb-40c0-950e-5a9901eb2030-kube-api-access-vm2ht\" (UniqueName: \"kubernetes.io/projected/be38df56-d9bb-40c0-950e-5a9901eb2030-kube-api-access-vm2ht\") pod \"be38df56-d9bb-40c0-950e-5a9901eb2030\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") "
Oct 30 12:58:07.321375 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.321293    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-openshift-ca\") pod \"be38df56-d9bb-40c0-950e-5a9901eb2030\" (UID: \"be38df56-d9bb-40c0-950e-5a9901eb2030\") "
Oct 30 12:58:07.321751 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.321719    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-openshift-ca" pod "be38df56-d9bb-40c0-950e-5a9901eb2030" (UID: "be38df56-d9bb-40c0-950e-5a9901eb2030"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:07.321873 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.321795    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.323558 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.323532    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be38df56-d9bb-40c0-950e-5a9901eb2030-kube-api-access-vm2ht" pod "be38df56-d9bb-40c0-950e-5a9901eb2030" (UID: "be38df56-d9bb-40c0-950e-5a9901eb2030"). InnerVolumeSpecName "kube-api-access-vm2ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:07.323791 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.323765    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-xks-ca" pod "be38df56-d9bb-40c0-950e-5a9901eb2030" (UID: "be38df56-d9bb-40c0-950e-5a9901eb2030"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:07.423729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.423677    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be38df56-d9bb-40c0-950e-5a9901eb2030-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.423729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.423724    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm2ht\" (UniqueName: \"kubernetes.io/projected/be38df56-d9bb-40c0-950e-5a9901eb2030-kube-api-access-vm2ht\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:07.952570 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.952529    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"]
Oct 30 12:58:07.953277 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.953260    4993 cpu_manager.go:414] "containerMap: removing container" podUID="be38df56-d9bb-40c0-950e-5a9901eb2030" containerName="revoke-keys"
Oct 30 12:58:07.953322 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.953279    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be38df56-d9bb-40c0-950e-5a9901eb2030" containerName="revoke-keys"
Oct 30 12:58:07.953405 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.953394    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="be38df56-d9bb-40c0-950e-5a9901eb2030" containerName="revoke-keys"
Oct 30 12:58:07.957395 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.957335    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:07.977612 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.977579    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"
Oct 30 12:58:07.977785 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.977579    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq" event={"ID":"be38df56-d9bb-40c0-950e-5a9901eb2030","Type":"ContainerDied","Data":"e4e4b1756b386dc88042f0dbc250b798e74e8204150bf7d17010bc205a813a5d"}
Oct 30 12:58:07.977785 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.977733    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4e4b1756b386dc88042f0dbc250b798e74e8204150bf7d17010bc205a813a5d"
Oct 30 12:58:07.981760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:07.981730    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"]
Oct 30 12:58:08.030017 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.029976    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.030243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.030037    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.030243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.030147    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdldh\" (UniqueName: \"kubernetes.io/projected/59396606-0545-4936-a10a-d5ca3cd99368-kube-api-access-pdldh\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.130772 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.130735    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.130978 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.130798    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdldh\" (UniqueName: \"kubernetes.io/projected/59396606-0545-4936-a10a-d5ca3cd99368-kube-api-access-pdldh\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.130978 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.130969    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.131614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.131593    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.133421 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.133397    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.156159 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.156127    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdldh\" (UniqueName: \"kubernetes.io/projected/59396606-0545-4936-a10a-d5ca3cd99368-kube-api-access-pdldh\") pod \"maas-api-revoke-sub-fc132f0d-ll7tv\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.274302 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.274260    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:08.590641 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:08.590607    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59396606_0545_4936_a10a_d5ca3cd99368.slice/crio-f082ec103c9cd27b59ee3df3efd78ec61e7bdb08d8e9c29cb96b95145b0a55c9 WatchSource:0}: Error finding container f082ec103c9cd27b59ee3df3efd78ec61e7bdb08d8e9c29cb96b95145b0a55c9: Status 404 returned error can't find the container with id f082ec103c9cd27b59ee3df3efd78ec61e7bdb08d8e9c29cb96b95145b0a55c9
Oct 30 12:58:08.596054 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.591293    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"]
Oct 30 12:58:08.596292 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:58:08.596175    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:58:08.987056 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.987015    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv" event={"ID":"59396606-0545-4936-a10a-d5ca3cd99368","Type":"ContainerStarted","Data":"9c080ed93d17167206fcef3d5b800c8fd6067ed56182c69b993f8e2f0175967f"}
Oct 30 12:58:08.987285 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:08.987084    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv" event={"ID":"59396606-0545-4936-a10a-d5ca3cd99368","Type":"ContainerStarted","Data":"f082ec103c9cd27b59ee3df3efd78ec61e7bdb08d8e9c29cb96b95145b0a55c9"}
Oct 30 12:58:09.031506 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:09.031425    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv" podStartSLOduration=2.031397058 podStartE2EDuration="2.031397058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:09.030480484 +0000 UTC m=+1553.506200098" watchObservedRunningTime="2026-10-30 12:58:09.031397058 +0000 UTC m=+1553.507116669"
Oct 30 12:58:09.998200 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:09.998167    4993 generic.go:356] "Generic (PLEG): container finished" podID="59396606-0545-4936-a10a-d5ca3cd99368" containerID="9c080ed93d17167206fcef3d5b800c8fd6067ed56182c69b993f8e2f0175967f" exitCode=0
Oct 30 12:58:09.998711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:09.998258    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv" event={"ID":"59396606-0545-4936-a10a-d5ca3cd99368","Type":"ContainerDied","Data":"9c080ed93d17167206fcef3d5b800c8fd6067ed56182c69b993f8e2f0175967f"}
Oct 30 12:58:11.179714 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.179680    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:11.260742 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.260645    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-openshift-ca\") pod \"59396606-0545-4936-a10a-d5ca3cd99368\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") "
Oct 30 12:58:11.260925 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.260749    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-xks-ca\") pod \"59396606-0545-4936-a10a-d5ca3cd99368\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") "
Oct 30 12:58:11.260925 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.260803    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59396606-0545-4936-a10a-d5ca3cd99368-kube-api-access-pdldh\" (UniqueName: \"kubernetes.io/projected/59396606-0545-4936-a10a-d5ca3cd99368-kube-api-access-pdldh\") pod \"59396606-0545-4936-a10a-d5ca3cd99368\" (UID: \"59396606-0545-4936-a10a-d5ca3cd99368\") "
Oct 30 12:58:11.261063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.261041    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-openshift-ca" pod "59396606-0545-4936-a10a-d5ca3cd99368" (UID: "59396606-0545-4936-a10a-d5ca3cd99368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:11.261253 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.261230    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:11.263192 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.263160    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-xks-ca" pod "59396606-0545-4936-a10a-d5ca3cd99368" (UID: "59396606-0545-4936-a10a-d5ca3cd99368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:11.263525 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.263487    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59396606-0545-4936-a10a-d5ca3cd99368-kube-api-access-pdldh" pod "59396606-0545-4936-a10a-d5ca3cd99368" (UID: "59396606-0545-4936-a10a-d5ca3cd99368"). InnerVolumeSpecName "kube-api-access-pdldh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:11.362090 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.362050    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59396606-0545-4936-a10a-d5ca3cd99368-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:11.362090 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:11.362084    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdldh\" (UniqueName: \"kubernetes.io/projected/59396606-0545-4936-a10a-d5ca3cd99368-kube-api-access-pdldh\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:12.013924 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:12.013887    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv" event={"ID":"59396606-0545-4936-a10a-d5ca3cd99368","Type":"ContainerDied","Data":"f082ec103c9cd27b59ee3df3efd78ec61e7bdb08d8e9c29cb96b95145b0a55c9"}
Oct 30 12:58:12.014162 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:12.013942    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f082ec103c9cd27b59ee3df3efd78ec61e7bdb08d8e9c29cb96b95145b0a55c9"
Oct 30 12:58:12.014162 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:12.013900    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"
Oct 30 12:58:16.783743 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.782507    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"]
Oct 30 12:58:16.785588 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.785559    4993 cpu_manager.go:414] "containerMap: removing container" podUID="59396606-0545-4936-a10a-d5ca3cd99368" containerName="revoke-keys"
Oct 30 12:58:16.785588 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.785585    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59396606-0545-4936-a10a-d5ca3cd99368" containerName="revoke-keys"
Oct 30 12:58:16.785779 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.785684    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="59396606-0545-4936-a10a-d5ca3cd99368" containerName="revoke-keys"
Oct 30 12:58:16.791155 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.791133    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:16.808007 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.807937    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"]
Oct 30 12:58:16.813236 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.812588    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:16.813236 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.812788    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:16.850777 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.850740    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:16.851548 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.851520    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:58:16.852401 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.852374    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:16.852549 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.852531    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xb2\" (UniqueName: \"kubernetes.io/projected/76feb2a3-41e7-4718-8980-af66eb089179-kube-api-access-c7xb2\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:16.852625 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.852582    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:16.954240 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.953677    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xb2\" (UniqueName: \"kubernetes.io/projected/76feb2a3-41e7-4718-8980-af66eb089179-kube-api-access-c7xb2\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:16.954240 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.953786    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:16.954240 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.953882    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:16.962237 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.955627    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:16.972242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:16.970686    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:17.007232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:17.003500    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xb2\" (UniqueName: \"kubernetes.io/projected/76feb2a3-41e7-4718-8980-af66eb089179-kube-api-access-c7xb2\") pod \"maas-api-revoke-sub-c0e24fae-h26mg\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:17.124289 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:17.120409    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:17.843238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:17.841343    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"]
Oct 30 12:58:18.162302 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:18.160121    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg" event={"ID":"76feb2a3-41e7-4718-8980-af66eb089179","Type":"ContainerStarted","Data":"67eca59ba5230460f4d5621b7d3dd61021e1343b24ce150bfe6404ed5f8ea15b"}
Oct 30 12:58:19.170920 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:19.170814    4993 generic.go:356] "Generic (PLEG): container finished" podID="76feb2a3-41e7-4718-8980-af66eb089179" containerID="6298ba842145b6af8690ddcd3598ea7c4f63a9396ccb64993fc8fb426bdbc506" exitCode=0
Oct 30 12:58:19.170920 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:19.170899    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg" event={"ID":"76feb2a3-41e7-4718-8980-af66eb089179","Type":"ContainerDied","Data":"6298ba842145b6af8690ddcd3598ea7c4f63a9396ccb64993fc8fb426bdbc506"}
Oct 30 12:58:20.020674 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.020633    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"]
Oct 30 12:58:20.024707 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.024680    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.045268 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.045222    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"]
Oct 30 12:58:20.051597 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.051565    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-tmp-dir\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.051781 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.051615    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-home\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.051865 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.051838    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-model-cache\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.051915 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.051891    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-dshm\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.052002 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.051982    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c48a685-e866-47d9-8673-673bb5bd0be3-tls-certs\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.052092 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.052073    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8fbm\" (UniqueName: \"kubernetes.io/projected/9c48a685-e866-47d9-8673-673bb5bd0be3-kube-api-access-p8fbm\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.055441 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.055403    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a8b03\"/\"distinct-2-w0-7a8b03-kserve-self-signed-certs\""
Oct 30 12:58:20.153513 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.153475    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-home\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.153722 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.153589    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-model-cache\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.153722 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.153631    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-dshm\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.153722 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.153670    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c48a685-e866-47d9-8673-673bb5bd0be3-tls-certs\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.153903 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.153756    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fbm\" (UniqueName: \"kubernetes.io/projected/9c48a685-e866-47d9-8673-673bb5bd0be3-kube-api-access-p8fbm\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.153903 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.153853    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-tmp-dir\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.154050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.154025    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-home\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.154286 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.154261    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-tmp-dir\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.154638 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.154614    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-model-cache\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.156901 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.156875    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-dshm\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.157449 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.157427    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c48a685-e866-47d9-8673-673bb5bd0be3-tls-certs\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.185711 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.185676    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8fbm\" (UniqueName: \"kubernetes.io/projected/9c48a685-e866-47d9-8673-673bb5bd0be3-kube-api-access-p8fbm\") pod \"distinct-2-w0-7a8b03-kserve-56d8776969-nr45s\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") " pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.342334 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.340530    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:20.449643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.449045    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:20.463891 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.463832    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-xks-ca\") pod \"76feb2a3-41e7-4718-8980-af66eb089179\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") "
Oct 30 12:58:20.464112 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.464045    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76feb2a3-41e7-4718-8980-af66eb089179-kube-api-access-c7xb2\" (UniqueName: \"kubernetes.io/projected/76feb2a3-41e7-4718-8980-af66eb089179-kube-api-access-c7xb2\") pod \"76feb2a3-41e7-4718-8980-af66eb089179\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") "
Oct 30 12:58:20.464222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.464184    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-openshift-ca\") pod \"76feb2a3-41e7-4718-8980-af66eb089179\" (UID: \"76feb2a3-41e7-4718-8980-af66eb089179\") "
Oct 30 12:58:20.470258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.467861    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-xks-ca" pod "76feb2a3-41e7-4718-8980-af66eb089179" (UID: "76feb2a3-41e7-4718-8980-af66eb089179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:20.470258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.468142    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:20.470258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.468513    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76feb2a3-41e7-4718-8980-af66eb089179-kube-api-access-c7xb2" pod "76feb2a3-41e7-4718-8980-af66eb089179" (UID: "76feb2a3-41e7-4718-8980-af66eb089179"). InnerVolumeSpecName "kube-api-access-c7xb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:20.470258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.468949    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-openshift-ca" pod "76feb2a3-41e7-4718-8980-af66eb089179" (UID: "76feb2a3-41e7-4718-8980-af66eb089179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:20.569954 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.569907    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xb2\" (UniqueName: \"kubernetes.io/projected/76feb2a3-41e7-4718-8980-af66eb089179-kube-api-access-c7xb2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:20.569954 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.569956    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76feb2a3-41e7-4718-8980-af66eb089179-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:20.704574 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.704537    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"]
Oct 30 12:58:20.707509 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:20.707462    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c48a685_e866_47d9_8673_673bb5bd0be3.slice/crio-1842e3e50d7a2e45321474bef409216fe989e25c9a70a8d219c6a35183d27d91 WatchSource:0}: Error finding container 1842e3e50d7a2e45321474bef409216fe989e25c9a70a8d219c6a35183d27d91: Status 404 returned error can't find the container with id 1842e3e50d7a2e45321474bef409216fe989e25c9a70a8d219c6a35183d27d91
Oct 30 12:58:20.989879 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.989834    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"]
Oct 30 12:58:20.990643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.990619    4993 cpu_manager.go:414] "containerMap: removing container" podUID="76feb2a3-41e7-4718-8980-af66eb089179" containerName="revoke-keys"
Oct 30 12:58:20.990741 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.990645    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76feb2a3-41e7-4718-8980-af66eb089179" containerName="revoke-keys"
Oct 30 12:58:20.990831 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.990819    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="76feb2a3-41e7-4718-8980-af66eb089179" containerName="revoke-keys"
Oct 30 12:58:20.995050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:20.995021    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.018565 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.018527    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"]
Oct 30 12:58:21.076625 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.076566    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8586284e-14f9-4b07-955e-3054491dd750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.076625 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.076631    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8586284e-14f9-4b07-955e-3054491dd750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.076897 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.076871    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhrg\" (UniqueName: \"kubernetes.io/projected/8586284e-14f9-4b07-955e-3054491dd750-kube-api-access-fdhrg\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.177803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.177759    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhrg\" (UniqueName: \"kubernetes.io/projected/8586284e-14f9-4b07-955e-3054491dd750-kube-api-access-fdhrg\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.178068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.177913    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8586284e-14f9-4b07-955e-3054491dd750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.178068 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.177942    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8586284e-14f9-4b07-955e-3054491dd750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.178926 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.178890    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8586284e-14f9-4b07-955e-3054491dd750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.184762 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.184723    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8586284e-14f9-4b07-955e-3054491dd750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.196959 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.196917    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" event={"ID":"9c48a685-e866-47d9-8673-673bb5bd0be3","Type":"ContainerStarted","Data":"df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876"}
Oct 30 12:58:21.197724 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.197694    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" event={"ID":"9c48a685-e866-47d9-8673-673bb5bd0be3","Type":"ContainerStarted","Data":"1842e3e50d7a2e45321474bef409216fe989e25c9a70a8d219c6a35183d27d91"}
Oct 30 12:58:21.198977 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.198951    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg" event={"ID":"76feb2a3-41e7-4718-8980-af66eb089179","Type":"ContainerDied","Data":"67eca59ba5230460f4d5621b7d3dd61021e1343b24ce150bfe6404ed5f8ea15b"}
Oct 30 12:58:21.199118 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.198994    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67eca59ba5230460f4d5621b7d3dd61021e1343b24ce150bfe6404ed5f8ea15b"
Oct 30 12:58:21.199118 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.199046    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"
Oct 30 12:58:21.207063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.207024    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhrg\" (UniqueName: \"kubernetes.io/projected/8586284e-14f9-4b07-955e-3054491dd750-kube-api-access-fdhrg\") pod \"maas-api-revoke-sub-a9bc41ff-5wlsf\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.245668 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.245557    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podStartSLOduration=2.245541694 podStartE2EDuration="2.245541694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:21.244866824 +0000 UTC m=+1565.720586436" watchObservedRunningTime="2026-10-30 12:58:21.245541694 +0000 UTC m=+1565.721261303"
Oct 30 12:58:21.311188 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.311146    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:21.341440 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.341405    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:21.662049 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:21.662017    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8586284e_14f9_4b07_955e_3054491dd750.slice/crio-77bdf07576ff8f762cc3ce0795e84e4a61ea1cd946c1fce22c1f08a3aba739cc WatchSource:0}: Error finding container 77bdf07576ff8f762cc3ce0795e84e4a61ea1cd946c1fce22c1f08a3aba739cc: Status 404 returned error can't find the container with id 77bdf07576ff8f762cc3ce0795e84e4a61ea1cd946c1fce22c1f08a3aba739cc
Oct 30 12:58:21.662682 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:21.662654    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"]
Oct 30 12:58:22.209060 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.209018    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf" event={"ID":"8586284e-14f9-4b07-955e-3054491dd750","Type":"ContainerStarted","Data":"22ab588898ae9db944c436ce08e4cb1a5d42a364c7f505a9d979fd0fdd8af38d"}
Oct 30 12:58:22.209574 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.209105    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf" event={"ID":"8586284e-14f9-4b07-955e-3054491dd750","Type":"ContainerStarted","Data":"77bdf07576ff8f762cc3ce0795e84e4a61ea1cd946c1fce22c1f08a3aba739cc"}
Oct 30 12:58:22.269494 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.269413    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf" podStartSLOduration=2.269396031 podStartE2EDuration="2.269396031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:22.263806998 +0000 UTC m=+1566.739526611" watchObservedRunningTime="2026-10-30 12:58:22.269396031 +0000 UTC m=+1566.745115681"
Oct 30 12:58:22.454238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.454127    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"]
Oct 30 12:58:22.458670 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.458640    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.486122 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.486081    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"]
Oct 30 12:58:22.498175 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.498139    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhbh\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-kube-api-access-jdhbh\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.498175 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.498184    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.498528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.498245    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.498528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.498268    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.498528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.498468    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.498673 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.498545    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.499025 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.499002    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.499148 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.499042    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.499148 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.499096    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb04db8-4e1d-4234-9519-647b5ac606d0-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.600303 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600248    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.600528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600388    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.600528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600445    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.600528 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600474    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600527    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb04db8-4e1d-4234-9519-647b5ac606d0-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600612    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhbh\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-kube-api-access-jdhbh\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600650    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600680    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600713    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.600738    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.601029    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601455 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.601085    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601455 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.601345    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.601754 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.601733    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb04db8-4e1d-4234-9519-647b5ac606d0-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.603860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.603824    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.604038 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.604021    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.625511 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.625479    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.626012 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.625988    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhbh\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-kube-api-access-jdhbh\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:22.778869 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:22.778828    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:23.101932 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:23.101899    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb04db8_4e1d_4234_9519_647b5ac606d0.slice/crio-8ba0703180044f21824dbd8e533a6fa5b2ff9a3116f4a802e3e9ec1cda1aab37 WatchSource:0}: Error finding container 8ba0703180044f21824dbd8e533a6fa5b2ff9a3116f4a802e3e9ec1cda1aab37: Status 404 returned error can't find the container with id 8ba0703180044f21824dbd8e533a6fa5b2ff9a3116f4a802e3e9ec1cda1aab37
Oct 30 12:58:23.104358 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.104325    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"]
Oct 30 12:58:23.105021 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.104980    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:23.105121 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.105068    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:23.105121 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.105106    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:23.222820 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.222784    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" event={"ID":"6fb04db8-4e1d-4234-9519-647b5ac606d0","Type":"ContainerStarted","Data":"150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039"}
Oct 30 12:58:23.223285 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.222850    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" event={"ID":"6fb04db8-4e1d-4234-9519-647b5ac606d0","Type":"ContainerStarted","Data":"8ba0703180044f21824dbd8e533a6fa5b2ff9a3116f4a802e3e9ec1cda1aab37"}
Oct 30 12:58:23.224817 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.224787    4993 generic.go:356] "Generic (PLEG): container finished" podID="8586284e-14f9-4b07-955e-3054491dd750" containerID="22ab588898ae9db944c436ce08e4cb1a5d42a364c7f505a9d979fd0fdd8af38d" exitCode=0
Oct 30 12:58:23.224945 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.224855    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf" event={"ID":"8586284e-14f9-4b07-955e-3054491dd750","Type":"ContainerDied","Data":"22ab588898ae9db944c436ce08e4cb1a5d42a364c7f505a9d979fd0fdd8af38d"}
Oct 30 12:58:23.279350 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.279283    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" podStartSLOduration=1.279264354 podStartE2EDuration="1.279264354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:23.274526639 +0000 UTC m=+1567.750246265" watchObservedRunningTime="2026-10-30 12:58:23.279264354 +0000 UTC m=+1567.754983969"
Oct 30 12:58:23.779547 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.779498    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:23.884765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.884715    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"]
Oct 30 12:58:23.895965 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.895931    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.896152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.896018    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"]
Oct 30 12:58:23.919721 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.919682    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.919893 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.919732    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.919893 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.919816    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482043e-237d-4780-95eb-ee9dedf162bc-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.920012 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.919888    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.920012 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.919948    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgttz\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-kube-api-access-wgttz\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.920238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.920188    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482043e-237d-4780-95eb-ee9dedf162bc-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.920345 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.920323    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.920430 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.920413    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:23.920632 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:23.920610    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022260 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022197    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482043e-237d-4780-95eb-ee9dedf162bc-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022432 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022273    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022432 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022320    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgttz\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-kube-api-access-wgttz\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022432 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022384    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482043e-237d-4780-95eb-ee9dedf162bc-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022612 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022459    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022612 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022511    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022719 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022648    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022719 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022702    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022821 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022729    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022903 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022882    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.022957 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.022934    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.023106 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.023086    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482043e-237d-4780-95eb-ee9dedf162bc-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.023187 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.023129    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.023187 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.023151    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.024987 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.024965    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.025351 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.025330    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482043e-237d-4780-95eb-ee9dedf162bc-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.051062 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.050993    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.051245 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.051200    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgttz\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-kube-api-access-wgttz\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.223900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.223858    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:24.518181 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.518154    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:24.605238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.605191    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"]
Oct 30 12:58:24.606547 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.606510    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:24.606725 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.606700    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:24.606865 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.606751    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:24.632808 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.632769    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8586284e-14f9-4b07-955e-3054491dd750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8586284e-14f9-4b07-955e-3054491dd750-maas-api-openshift-ca\") pod \"8586284e-14f9-4b07-955e-3054491dd750\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") "
Oct 30 12:58:24.633480 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.633460    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8586284e-14f9-4b07-955e-3054491dd750-kube-api-access-fdhrg\" (UniqueName: \"kubernetes.io/projected/8586284e-14f9-4b07-955e-3054491dd750-kube-api-access-fdhrg\") pod \"8586284e-14f9-4b07-955e-3054491dd750\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") "
Oct 30 12:58:24.634067 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.634046    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8586284e-14f9-4b07-955e-3054491dd750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8586284e-14f9-4b07-955e-3054491dd750-maas-api-xks-ca\") pod \"8586284e-14f9-4b07-955e-3054491dd750\" (UID: \"8586284e-14f9-4b07-955e-3054491dd750\") "
Oct 30 12:58:24.635029 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.633318    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8586284e-14f9-4b07-955e-3054491dd750-maas-api-openshift-ca" pod "8586284e-14f9-4b07-955e-3054491dd750" (UID: "8586284e-14f9-4b07-955e-3054491dd750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:24.635292 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.635273    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8586284e-14f9-4b07-955e-3054491dd750-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:24.641032 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.641002    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8586284e-14f9-4b07-955e-3054491dd750-maas-api-xks-ca" pod "8586284e-14f9-4b07-955e-3054491dd750" (UID: "8586284e-14f9-4b07-955e-3054491dd750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:24.641140 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.641030    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8586284e-14f9-4b07-955e-3054491dd750-kube-api-access-fdhrg" pod "8586284e-14f9-4b07-955e-3054491dd750" (UID: "8586284e-14f9-4b07-955e-3054491dd750"). InnerVolumeSpecName "kube-api-access-fdhrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:24.737091 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.737059    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhrg\" (UniqueName: \"kubernetes.io/projected/8586284e-14f9-4b07-955e-3054491dd750-kube-api-access-fdhrg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:24.737091 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.737091    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8586284e-14f9-4b07-955e-3054491dd750-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:24.785395 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.785354    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 12:58:24.785581 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:24.785410    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 12:58:25.240542 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:25.240515    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"
Oct 30 12:58:25.241019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:25.240520    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf" event={"ID":"8586284e-14f9-4b07-955e-3054491dd750","Type":"ContainerDied","Data":"77bdf07576ff8f762cc3ce0795e84e4a61ea1cd946c1fce22c1f08a3aba739cc"}
Oct 30 12:58:25.241019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:25.240727    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77bdf07576ff8f762cc3ce0795e84e4a61ea1cd946c1fce22c1f08a3aba739cc"
Oct 30 12:58:25.243026 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:25.242998    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" event={"ID":"c482043e-237d-4780-95eb-ee9dedf162bc","Type":"ContainerStarted","Data":"98c06f3aeeb489af0da86c21bf2ab57e6614d083ede766b0458c4815a659de8f"}
Oct 30 12:58:25.243163 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:25.243049    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" event={"ID":"c482043e-237d-4780-95eb-ee9dedf162bc","Type":"ContainerStarted","Data":"492434dc951b3dc641802893fdb619cef7a7de591798797be2dafe57e45ec6d4"}
Oct 30 12:58:25.307510 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:25.307442    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podStartSLOduration=2.307410934 podStartE2EDuration="2.307410934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:25.296737199 +0000 UTC m=+1569.772456809" watchObservedRunningTime="2026-10-30 12:58:25.307410934 +0000 UTC m=+1569.783130545"
Oct 30 12:58:25.780232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:25.779486    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 12:58:25.780232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:25.779548    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 12:58:26.225045 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:26.224942    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:26.226267 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:26.226244    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 12:58:26.226321 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:26.226289    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 12:58:26.779756 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:26.779719    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 12:58:26.780359 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:26.780328    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 12:58:27.224806 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.224720    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 12:58:27.225044 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.225014    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 12:58:27.313729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.313679    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"]
Oct 30 12:58:27.314736 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.314706    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8586284e-14f9-4b07-955e-3054491dd750" containerName="revoke-keys"
Oct 30 12:58:27.314943 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.314927    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8586284e-14f9-4b07-955e-3054491dd750" containerName="revoke-keys"
Oct 30 12:58:27.317233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.315269    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8586284e-14f9-4b07-955e-3054491dd750" containerName="revoke-keys"
Oct 30 12:58:27.321526 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.321454    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.336159 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.336140    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"]
Oct 30 12:58:27.343707 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.343668    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:27.343707 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.343682    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:27.343832 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.343733    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:27.360228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.352180    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:58:27.408250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.391028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.408250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.392243    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pw52\" (UniqueName: \"kubernetes.io/projected/3f7dfbab-ae0f-417d-a935-f0b887dcb782-kube-api-access-2pw52\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.408250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.396498    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.500272 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.500228    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pw52\" (UniqueName: \"kubernetes.io/projected/3f7dfbab-ae0f-417d-a935-f0b887dcb782-kube-api-access-2pw52\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.500670 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.500644    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.500854 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.500834    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.519259 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.509798    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.521688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.521653    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.538642 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.538616    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pw52\" (UniqueName: \"kubernetes.io/projected/3f7dfbab-ae0f-417d-a935-f0b887dcb782-kube-api-access-2pw52\") pod \"maas-api-revoke-sub-b9239531-58glr\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.637700 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.637664    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:27.785229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.779650    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 12:58:27.785229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:27.779709    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 12:58:28.026448 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:28.026377    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7dfbab_ae0f_417d_a935_f0b887dcb782.slice/crio-74ee6e4e3851cc81cb569921b4d48bdc162080069d0b4f68b2c62def95eefe63 WatchSource:0}: Error finding container 74ee6e4e3851cc81cb569921b4d48bdc162080069d0b4f68b2c62def95eefe63: Status 404 returned error can't find the container with id 74ee6e4e3851cc81cb569921b4d48bdc162080069d0b4f68b2c62def95eefe63
Oct 30 12:58:28.027861 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:28.027832    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"]
Oct 30 12:58:28.224582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:28.224548    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 12:58:28.224762 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:28.224609    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 12:58:28.314287 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:28.314023    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr" event={"ID":"3f7dfbab-ae0f-417d-a935-f0b887dcb782","Type":"ContainerStarted","Data":"74ee6e4e3851cc81cb569921b4d48bdc162080069d0b4f68b2c62def95eefe63"}
Oct 30 12:58:28.780234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:28.779655    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 12:58:28.780234 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:28.779719    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 12:58:29.225269 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:29.225230    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 12:58:29.225638 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:29.225297    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 12:58:29.322469 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:29.322439    4993 generic.go:356] "Generic (PLEG): container finished" podID="3f7dfbab-ae0f-417d-a935-f0b887dcb782" containerID="50b9619144fddc0ebdefe6bab3fa4686b5b03a7a94b41e7f92b1162dbe3e647e" exitCode=0
Oct 30 12:58:29.322645 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:29.322523    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr" event={"ID":"3f7dfbab-ae0f-417d-a935-f0b887dcb782","Type":"ContainerDied","Data":"50b9619144fddc0ebdefe6bab3fa4686b5b03a7a94b41e7f92b1162dbe3e647e"}
Oct 30 12:58:29.782598 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:29.782560    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 12:58:29.782845 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:29.782818    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 12:58:30.224307 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.224185    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 12:58:30.224307 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.224275    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 12:58:30.342001 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.341964    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:30.405493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.405460    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:30.635960 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.635924    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:30.815242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.811742    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-openshift-ca\") pod \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") "
Oct 30 12:58:30.815242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.811937    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7dfbab-ae0f-417d-a935-f0b887dcb782-kube-api-access-2pw52\" (UniqueName: \"kubernetes.io/projected/3f7dfbab-ae0f-417d-a935-f0b887dcb782-kube-api-access-2pw52\") pod \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") "
Oct 30 12:58:30.815242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.812045    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-xks-ca\") pod \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\" (UID: \"3f7dfbab-ae0f-417d-a935-f0b887dcb782\") "
Oct 30 12:58:30.815242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.813178    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-openshift-ca" pod "3f7dfbab-ae0f-417d-a935-f0b887dcb782" (UID: "3f7dfbab-ae0f-417d-a935-f0b887dcb782"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:30.820241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.819981    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-xks-ca" pod "3f7dfbab-ae0f-417d-a935-f0b887dcb782" (UID: "3f7dfbab-ae0f-417d-a935-f0b887dcb782"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:30.820241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.820092    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7dfbab-ae0f-417d-a935-f0b887dcb782-kube-api-access-2pw52" pod "3f7dfbab-ae0f-417d-a935-f0b887dcb782" (UID: "3f7dfbab-ae0f-417d-a935-f0b887dcb782"). InnerVolumeSpecName "kube-api-access-2pw52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:30.913963 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.913869    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pw52\" (UniqueName: \"kubernetes.io/projected/3f7dfbab-ae0f-417d-a935-f0b887dcb782-kube-api-access-2pw52\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:30.913963 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.913899    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:30.913963 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:30.913912    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7dfbab-ae0f-417d-a935-f0b887dcb782-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:31.043522 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.043487    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:31.044046 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.044014    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:31.045274 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.045251    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 12:58:31.225304 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.225221    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 12:58:31.225304 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.225291    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 12:58:31.340151 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.340124    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"
Oct 30 12:58:31.342425 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.342385    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr" event={"ID":"3f7dfbab-ae0f-417d-a935-f0b887dcb782","Type":"ContainerDied","Data":"74ee6e4e3851cc81cb569921b4d48bdc162080069d0b4f68b2c62def95eefe63"}
Oct 30 12:58:31.342924 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.342458    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74ee6e4e3851cc81cb569921b4d48bdc162080069d0b4f68b2c62def95eefe63"
Oct 30 12:58:31.356547 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.356485    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:58:31.521970 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.521930    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"]
Oct 30 12:58:31.522471 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.522456    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3f7dfbab-ae0f-417d-a935-f0b887dcb782" containerName="revoke-keys"
Oct 30 12:58:31.522521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.522472    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7dfbab-ae0f-417d-a935-f0b887dcb782" containerName="revoke-keys"
Oct 30 12:58:31.522569 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.522560    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7dfbab-ae0f-417d-a935-f0b887dcb782" containerName="revoke-keys"
Oct 30 12:58:31.528842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.527995    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.546840 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.546807    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:31.547011 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.546879    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:58:31.547011 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.547000    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:31.547688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.547669    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:31.552589 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.552566    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"]
Oct 30 12:58:31.629087 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.629050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wb6k\" (UniqueName: \"kubernetes.io/projected/c5705f87-acc0-4c1d-8d65-7294d7d77b63-kube-api-access-5wb6k\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.629423 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.629402    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.630232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.630193    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.731283 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.731223    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wb6k\" (UniqueName: \"kubernetes.io/projected/c5705f87-acc0-4c1d-8d65-7294d7d77b63-kube-api-access-5wb6k\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.731494 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.731296    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.731771 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.731739    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.732821 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.732688    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.734951 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.734605    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.761882 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.761818    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wb6k\" (UniqueName: \"kubernetes.io/projected/c5705f87-acc0-4c1d-8d65-7294d7d77b63-kube-api-access-5wb6k\") pod \"maas-api-revoke-sub-a3085f50-x6dpc\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:31.846245 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:31.846117    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:32.240867 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:32.240223    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:32.241958 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:32.241562    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:32.243127 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:32.243106    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 12:58:32.254443 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:32.254409    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"]
Oct 30 12:58:32.419393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:32.419356    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc" event={"ID":"c5705f87-acc0-4c1d-8d65-7294d7d77b63","Type":"ContainerStarted","Data":"7afa07f20bbd4f6251be8cb64c9acaf61390bb1856794241eb304ab31a4745b6"}
Oct 30 12:58:33.498231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:33.497837    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc" event={"ID":"c5705f87-acc0-4c1d-8d65-7294d7d77b63","Type":"ContainerStarted","Data":"40085b077bcb55a99768fdbffb3461e0b40a2423347d53a470893a5a85b90fd8"}
Oct 30 12:58:33.566232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:33.565505    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc" podStartSLOduration=2.565487 podStartE2EDuration="2.565487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:33.562943115 +0000 UTC m=+1578.038662728" watchObservedRunningTime="2026-10-30 12:58:33.565487 +0000 UTC m=+1578.041206611"
Oct 30 12:58:34.514232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:34.514177    4993 generic.go:356] "Generic (PLEG): container finished" podID="c5705f87-acc0-4c1d-8d65-7294d7d77b63" containerID="40085b077bcb55a99768fdbffb3461e0b40a2423347d53a470893a5a85b90fd8" exitCode=0
Oct 30 12:58:34.520232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:34.518339    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc" event={"ID":"c5705f87-acc0-4c1d-8d65-7294d7d77b63","Type":"ContainerDied","Data":"40085b077bcb55a99768fdbffb3461e0b40a2423347d53a470893a5a85b90fd8"}
Oct 30 12:58:35.717992 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.717965    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:35.854576 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.854474    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5705f87-acc0-4c1d-8d65-7294d7d77b63-kube-api-access-5wb6k\" (UniqueName: \"kubernetes.io/projected/c5705f87-acc0-4c1d-8d65-7294d7d77b63-kube-api-access-5wb6k\") pod \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") "
Oct 30 12:58:35.854761 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.854739    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-openshift-ca\") pod \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") "
Oct 30 12:58:35.854826 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.854797    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-xks-ca\") pod \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\" (UID: \"c5705f87-acc0-4c1d-8d65-7294d7d77b63\") "
Oct 30 12:58:35.855128 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.855099    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-openshift-ca" pod "c5705f87-acc0-4c1d-8d65-7294d7d77b63" (UID: "c5705f87-acc0-4c1d-8d65-7294d7d77b63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:35.855300 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.855276    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:35.856807 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.856774    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5705f87-acc0-4c1d-8d65-7294d7d77b63-kube-api-access-5wb6k" pod "c5705f87-acc0-4c1d-8d65-7294d7d77b63" (UID: "c5705f87-acc0-4c1d-8d65-7294d7d77b63"). InnerVolumeSpecName "kube-api-access-5wb6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:35.856907 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.856883    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-xks-ca" pod "c5705f87-acc0-4c1d-8d65-7294d7d77b63" (UID: "c5705f87-acc0-4c1d-8d65-7294d7d77b63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:35.956179 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.956143    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5705f87-acc0-4c1d-8d65-7294d7d77b63-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:35.956179 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:35.956177    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wb6k\" (UniqueName: \"kubernetes.io/projected/c5705f87-acc0-4c1d-8d65-7294d7d77b63-kube-api-access-5wb6k\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:36.394368 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.391506    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"]
Oct 30 12:58:36.394368 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.392704    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c5705f87-acc0-4c1d-8d65-7294d7d77b63" containerName="revoke-keys"
Oct 30 12:58:36.394368 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.392729    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5705f87-acc0-4c1d-8d65-7294d7d77b63" containerName="revoke-keys"
Oct 30 12:58:36.394368 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.392981    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5705f87-acc0-4c1d-8d65-7294d7d77b63" containerName="revoke-keys"
Oct 30 12:58:36.398595 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.398564    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.422810 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.422782    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"]
Oct 30 12:58:36.462371 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462332    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.462554 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462382    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.462554 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462414    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.462677 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462541    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.462677 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462589    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mg2\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-kube-api-access-47mg2\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.462677 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462625    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7857c6-5221-4532-ae92-fa36943e76b1-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.462821 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462695    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.462821 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462795    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.462915 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.462888    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.539456 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.539427    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"
Oct 30 12:58:36.539628 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.539454    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc" event={"ID":"c5705f87-acc0-4c1d-8d65-7294d7d77b63","Type":"ContainerDied","Data":"7afa07f20bbd4f6251be8cb64c9acaf61390bb1856794241eb304ab31a4745b6"}
Oct 30 12:58:36.539628 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.539509    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7afa07f20bbd4f6251be8cb64c9acaf61390bb1856794241eb304ab31a4745b6"
Oct 30 12:58:36.563544 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563513    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7857c6-5221-4532-ae92-fa36943e76b1-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.563675 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563573    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.563675 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563634    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.563765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563686    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.563765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563723    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.563765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563742    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.563765 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563762    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.563944 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563798    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.563944 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563830    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mg2\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-kube-api-access-47mg2\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.564016 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.563963    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.564113 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.564092    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.564319 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.564298    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7857c6-5221-4532-ae92-fa36943e76b1-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.564470 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.564450    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.564532 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.564485    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.566124 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.566102    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.566503 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.566480    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.587620 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.587589    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.587746 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.587728    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mg2\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-kube-api-access-47mg2\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:36.733151 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:36.733107    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:37.078948 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:37.078912    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"]
Oct 30 12:58:37.080811 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:37.080778    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7857c6_5221_4532_ae92_fa36943e76b1.slice/crio-6d7dbb2ab7d7893f67cd3c21b9e3b9f4e9b0eda397cc0d3d57b1f068e7b6a9e2 WatchSource:0}: Error finding container 6d7dbb2ab7d7893f67cd3c21b9e3b9f4e9b0eda397cc0d3d57b1f068e7b6a9e2: Status 404 returned error can't find the container with id 6d7dbb2ab7d7893f67cd3c21b9e3b9f4e9b0eda397cc0d3d57b1f068e7b6a9e2
Oct 30 12:58:37.083645 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:37.083605    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:37.083734 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:37.083691    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:37.083770 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:37.083732    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:58:37.550611 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:37.550563    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" event={"ID":"cb7857c6-5221-4532-ae92-fa36943e76b1","Type":"ContainerStarted","Data":"8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f"}
Oct 30 12:58:37.550799 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:37.550629    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" event={"ID":"cb7857c6-5221-4532-ae92-fa36943e76b1","Type":"ContainerStarted","Data":"6d7dbb2ab7d7893f67cd3c21b9e3b9f4e9b0eda397cc0d3d57b1f068e7b6a9e2"}
Oct 30 12:58:37.613705 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:37.613645    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podStartSLOduration=1.613627323 podStartE2EDuration="1.613627323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:37.611368544 +0000 UTC m=+1582.087088155" watchObservedRunningTime="2026-10-30 12:58:37.613627323 +0000 UTC m=+1582.089346932"
Oct 30 12:58:37.733333 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:37.733292    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:38.599146 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:58:38.599099    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:58:38.736241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:38.736176    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 12:58:38.736643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:38.736252    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 12:58:39.733842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:39.733799    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 12:58:39.734005 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:39.733867    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 12:58:40.739655 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:40.739615    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 12:58:40.740293 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:40.740255    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 12:58:41.733789 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:41.733751    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 12:58:41.733962 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:41.733824    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 12:58:42.009486 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.009398    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"]
Oct 30 12:58:42.013734 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.013707    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.028988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.028959    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:42.040921 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.040885    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:42.041116 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.040892    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:42.041116 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.041085    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:58:42.045673 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.045640    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"]
Oct 30 12:58:42.180229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.180049    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.180229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.180158    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88p8\" (UniqueName: \"kubernetes.io/projected/d9b9e9d7-d593-4abf-bac3-f72c72d69590-kube-api-access-x88p8\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.180634 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.180254    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.289283 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.288255    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.289283 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.289093    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.289598 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.289291    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88p8\" (UniqueName: \"kubernetes.io/projected/d9b9e9d7-d593-4abf-bac3-f72c72d69590-kube-api-access-x88p8\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.295258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.289760    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.314335 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.312980    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.316841 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.316790    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88p8\" (UniqueName: \"kubernetes.io/projected/d9b9e9d7-d593-4abf-bac3-f72c72d69590-kube-api-access-x88p8\") pod \"maas-api-revoke-sub-f1316aad-llmpw\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.328086 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.328048    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:42.690801 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:42.690761    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b9e9d7_d593_4abf_bac3_f72c72d69590.slice/crio-b38b2c38ccbbdcda920380ef4bb291da91673416979f40453c3d46de318534b1 WatchSource:0}: Error finding container b38b2c38ccbbdcda920380ef4bb291da91673416979f40453c3d46de318534b1: Status 404 returned error can't find the container with id b38b2c38ccbbdcda920380ef4bb291da91673416979f40453c3d46de318534b1
Oct 30 12:58:42.692112 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.692080    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"]
Oct 30 12:58:42.733944 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.733605    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 12:58:42.733944 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:42.733669    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 12:58:43.638285 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:43.638245    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw" event={"ID":"d9b9e9d7-d593-4abf-bac3-f72c72d69590","Type":"ContainerStarted","Data":"5430af644e85fb1a687404219d83cfe21e994993a777714f3b4fc3ad45a44422"}
Oct 30 12:58:43.638808 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:43.638345    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw" event={"ID":"d9b9e9d7-d593-4abf-bac3-f72c72d69590","Type":"ContainerStarted","Data":"b38b2c38ccbbdcda920380ef4bb291da91673416979f40453c3d46de318534b1"}
Oct 30 12:58:43.682613 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:43.682542    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw" podStartSLOduration=2.682521999 podStartE2EDuration="2.682521999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:43.680039273 +0000 UTC m=+1588.155758894" watchObservedRunningTime="2026-10-30 12:58:43.682521999 +0000 UTC m=+1588.158241612"
Oct 30 12:58:43.734501 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:43.734461    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 12:58:43.734673 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:43.734532    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 12:58:44.652118 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:44.652070    4993 generic.go:356] "Generic (PLEG): container finished" podID="d9b9e9d7-d593-4abf-bac3-f72c72d69590" containerID="5430af644e85fb1a687404219d83cfe21e994993a777714f3b4fc3ad45a44422" exitCode=0
Oct 30 12:58:44.652644 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:44.652141    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw" event={"ID":"d9b9e9d7-d593-4abf-bac3-f72c72d69590","Type":"ContainerDied","Data":"5430af644e85fb1a687404219d83cfe21e994993a777714f3b4fc3ad45a44422"}
Oct 30 12:58:44.738222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:44.738182    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:44.738616 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:44.738593    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:44.739241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:44.739200    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 12:58:45.362717 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.362679    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"]
Oct 30 12:58:45.367072 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.367044    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.388082 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.388048    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"]
Oct 30 12:58:45.431383 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.431336    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.431588 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.431405    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jbb\" (UniqueName: \"kubernetes.io/projected/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-kube-api-access-b7jbb\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.431588 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.431511    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.532582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.532544    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.532582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.532589    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jbb\" (UniqueName: \"kubernetes.io/projected/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-kube-api-access-b7jbb\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.532850 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.532608    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.533318 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.533294    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.535132 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.535115    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.556499 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.556460    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jbb\" (UniqueName: \"kubernetes.io/projected/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-kube-api-access-b7jbb\") pod \"maas-api-revoke-sub-df8a04de-k97gh\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.682504 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.682465    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:45.902848 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.902811    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:45.980472 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.979821    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-xks-ca\") pod \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") "
Oct 30 12:58:45.980472 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.979900    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-openshift-ca\") pod \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") "
Oct 30 12:58:45.980472 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.979932    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b9e9d7-d593-4abf-bac3-f72c72d69590-kube-api-access-x88p8\" (UniqueName: \"kubernetes.io/projected/d9b9e9d7-d593-4abf-bac3-f72c72d69590-kube-api-access-x88p8\") pod \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\" (UID: \"d9b9e9d7-d593-4abf-bac3-f72c72d69590\") "
Oct 30 12:58:45.980885 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.980852    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-openshift-ca" pod "d9b9e9d7-d593-4abf-bac3-f72c72d69590" (UID: "d9b9e9d7-d593-4abf-bac3-f72c72d69590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:45.983268 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.983175    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-xks-ca" pod "d9b9e9d7-d593-4abf-bac3-f72c72d69590" (UID: "d9b9e9d7-d593-4abf-bac3-f72c72d69590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:45.993661 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:45.993625    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b9e9d7-d593-4abf-bac3-f72c72d69590-kube-api-access-x88p8" pod "d9b9e9d7-d593-4abf-bac3-f72c72d69590" (UID: "d9b9e9d7-d593-4abf-bac3-f72c72d69590"). InnerVolumeSpecName "kube-api-access-x88p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:46.033048 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:46.033002    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7e6fdf_d388_4793_aa93_5c7055c7ab07.slice/crio-f85be2774d0c6bd2edfa7595e46990e0f29779b35c52b8966951089e722cc752 WatchSource:0}: Error finding container f85be2774d0c6bd2edfa7595e46990e0f29779b35c52b8966951089e722cc752: Status 404 returned error can't find the container with id f85be2774d0c6bd2edfa7595e46990e0f29779b35c52b8966951089e722cc752
Oct 30 12:58:46.033423 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.033394    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"]
Oct 30 12:58:46.081624 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.081587    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:46.081769 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.081627    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x88p8\" (UniqueName: \"kubernetes.io/projected/d9b9e9d7-d593-4abf-bac3-f72c72d69590-kube-api-access-x88p8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:46.081769 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.081647    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b9e9d7-d593-4abf-bac3-f72c72d69590-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:46.745329 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.745292    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw" event={"ID":"d9b9e9d7-d593-4abf-bac3-f72c72d69590","Type":"ContainerDied","Data":"b38b2c38ccbbdcda920380ef4bb291da91673416979f40453c3d46de318534b1"}
Oct 30 12:58:46.745958 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.745935    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b38b2c38ccbbdcda920380ef4bb291da91673416979f40453c3d46de318534b1"
Oct 30 12:58:46.746137 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.746124    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"
Oct 30 12:58:46.766837 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.753268    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh" event={"ID":"cf7e6fdf-d388-4793-aa93-5c7055c7ab07","Type":"ContainerStarted","Data":"ac36c9657e0940fd8a5ba68b452a13706931361c3f31ffaae690bfbcc085d8a1"}
Oct 30 12:58:46.766837 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.753327    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh" event={"ID":"cf7e6fdf-d388-4793-aa93-5c7055c7ab07","Type":"ContainerStarted","Data":"f85be2774d0c6bd2edfa7595e46990e0f29779b35c52b8966951089e722cc752"}
Oct 30 12:58:46.796104 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:46.796042    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh" podStartSLOduration=1.796021718 podStartE2EDuration="1.796021718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:46.794551795 +0000 UTC m=+1591.270271406" watchObservedRunningTime="2026-10-30 12:58:46.796021718 +0000 UTC m=+1591.271741332"
Oct 30 12:58:47.909109 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:47.909068    4993 generic.go:356] "Generic (PLEG): container finished" podID="cf7e6fdf-d388-4793-aa93-5c7055c7ab07" containerID="ac36c9657e0940fd8a5ba68b452a13706931361c3f31ffaae690bfbcc085d8a1" exitCode=0
Oct 30 12:58:47.909893 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:47.909861    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh" event={"ID":"cf7e6fdf-d388-4793-aa93-5c7055c7ab07","Type":"ContainerDied","Data":"ac36c9657e0940fd8a5ba68b452a13706931361c3f31ffaae690bfbcc085d8a1"}
Oct 30 12:58:48.574233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:48.573319    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:48.650232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:48.647303    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.280397 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.280341    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.324560 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.324496    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.452482 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.348837    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.473748 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.473302    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 12:58:49.489116 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.489073    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 12:58:49.660194 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.657492    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 12:58:49.660194 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.658342    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:58:49.956175 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.956068    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:49.957473 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.957442    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:58:49.965370 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.965336    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"
Oct 30 12:58:49.965778 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.965748    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh" event={"ID":"cf7e6fdf-d388-4793-aa93-5c7055c7ab07","Type":"ContainerDied","Data":"f85be2774d0c6bd2edfa7595e46990e0f29779b35c52b8966951089e722cc752"}
Oct 30 12:58:49.965957 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:49.965939    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f85be2774d0c6bd2edfa7595e46990e0f29779b35c52b8966951089e722cc752"
Oct 30 12:58:50.103096 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.103058    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-xks-ca\") pod \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") "
Oct 30 12:58:50.110508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.110479    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-openshift-ca\") pod \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") "
Oct 30 12:58:50.119557 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.119521    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-kube-api-access-b7jbb\" (UniqueName: \"kubernetes.io/projected/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-kube-api-access-b7jbb\") pod \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\" (UID: \"cf7e6fdf-d388-4793-aa93-5c7055c7ab07\") "
Oct 30 12:58:50.120760 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.117465    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-openshift-ca" pod "cf7e6fdf-d388-4793-aa93-5c7055c7ab07" (UID: "cf7e6fdf-d388-4793-aa93-5c7055c7ab07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:50.125072 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.125041    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-xks-ca" pod "cf7e6fdf-d388-4793-aa93-5c7055c7ab07" (UID: "cf7e6fdf-d388-4793-aa93-5c7055c7ab07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:50.133157 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.133120    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-kube-api-access-b7jbb" pod "cf7e6fdf-d388-4793-aa93-5c7055c7ab07" (UID: "cf7e6fdf-d388-4793-aa93-5c7055c7ab07"). InnerVolumeSpecName "kube-api-access-b7jbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:50.222452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.222292    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:50.222452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.222335    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:50.222452 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:50.222365    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jbb\" (UniqueName: \"kubernetes.io/projected/cf7e6fdf-d388-4793-aa93-5c7055c7ab07-kube-api-access-b7jbb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:51.020333 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.020292    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"]
Oct 30 12:58:51.020835 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.020820    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d9b9e9d7-d593-4abf-bac3-f72c72d69590" containerName="revoke-keys"
Oct 30 12:58:51.020835 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.020834    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b9e9d7-d593-4abf-bac3-f72c72d69590" containerName="revoke-keys"
Oct 30 12:58:51.020946 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.020843    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cf7e6fdf-d388-4793-aa93-5c7055c7ab07" containerName="revoke-keys"
Oct 30 12:58:51.020946 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.020848    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7e6fdf-d388-4793-aa93-5c7055c7ab07" containerName="revoke-keys"
Oct 30 12:58:51.020946 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.020938    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b9e9d7-d593-4abf-bac3-f72c72d69590" containerName="revoke-keys"
Oct 30 12:58:51.020946 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.020945    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7e6fdf-d388-4793-aa93-5c7055c7ab07" containerName="revoke-keys"
Oct 30 12:58:51.024830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.024806    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.045459 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.045423    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"]
Oct 30 12:58:51.047899 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.047867    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"distinct-2-w2-bb86fd-kserve-self-signed-certs\""
Oct 30 12:58:51.132803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.132762    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9wp\" (UniqueName: \"kubernetes.io/projected/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-kube-api-access-6w9wp\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.133002 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.132815    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-dshm\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.133002 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.132839    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-model-cache\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.133129 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.133063    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-home\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.133185 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.133109    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tmp-dir\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.133185 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.133163    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tls-certs\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.238251 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.238189    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9wp\" (UniqueName: \"kubernetes.io/projected/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-kube-api-access-6w9wp\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.238559 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.238535    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-dshm\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.238695 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.238679    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-model-cache\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.238924 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.238905    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-home\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.239507 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.239488    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tmp-dir\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.255231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.248369    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tls-certs\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.255231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.239397    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-model-cache\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.255231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.239883    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tmp-dir\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.255231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.239431    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-home\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.258826 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.258794    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tls-certs\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.275169 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.275088    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9wp\" (UniqueName: \"kubernetes.io/projected/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-kube-api-access-6w9wp\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.275655 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.275578    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-dshm\") pod \"distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.339521 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.339478    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:51.664588 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.664544    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"]
Oct 30 12:58:51.667031 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:51.666994    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddef5b5b9_fa1f_4d52_b32f_6ab32cbc552d.slice/crio-9de2137453069efdc1c583cef0f480612916aff232ea836a038720a0649f4d90 WatchSource:0}: Error finding container 9de2137453069efdc1c583cef0f480612916aff232ea836a038720a0649f4d90: Status 404 returned error can't find the container with id 9de2137453069efdc1c583cef0f480612916aff232ea836a038720a0649f4d90
Oct 30 12:58:51.983026 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.982979    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" event={"ID":"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d","Type":"ContainerStarted","Data":"7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf"}
Oct 30 12:58:51.983279 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:51.983039    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" event={"ID":"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d","Type":"ContainerStarted","Data":"9de2137453069efdc1c583cef0f480612916aff232ea836a038720a0649f4d90"}
Oct 30 12:58:52.030430 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.030371    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podStartSLOduration=2.030354759 podStartE2EDuration="2.030354759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:52.02827124 +0000 UTC m=+1596.503990851" watchObservedRunningTime="2026-10-30 12:58:52.030354759 +0000 UTC m=+1596.506074391"
Oct 30 12:58:52.339833 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.339737    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:58:52.415319 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.415259    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"]
Oct 30 12:58:52.419503 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.419472    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.437144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.437109    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:58:52.437478 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.437456    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:58:52.442833 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.442790    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"]
Oct 30 12:58:52.449449 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.448781    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:58:52.449449 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.448852    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:58:52.465759 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.465661    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.466003 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.465845    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdg9s\" (UniqueName: \"kubernetes.io/projected/7c46fa7a-df92-4fa8-a054-9de3c0620cca-kube-api-access-cdg9s\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.466073 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.466028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.567382 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.567341    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdg9s\" (UniqueName: \"kubernetes.io/projected/7c46fa7a-df92-4fa8-a054-9de3c0620cca-kube-api-access-cdg9s\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.567582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.567530    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.567693 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.567675    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.568706 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.568677    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.570178 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.570148    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.591238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.591146    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdg9s\" (UniqueName: \"kubernetes.io/projected/7c46fa7a-df92-4fa8-a054-9de3c0620cca-kube-api-access-cdg9s\") pod \"maas-api-revoke-sub-177c2301-fsw44\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:52.734399 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:52.734358    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:53.046614 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:53.046573    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"]
Oct 30 12:58:53.047053 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:53.047028    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c46fa7a_df92_4fa8_a054_9de3c0620cca.slice/crio-32d49d35c49cfeb9fa549d8be292cac349fe0eff4cc5b7f8785fe1f2267154e8 WatchSource:0}: Error finding container 32d49d35c49cfeb9fa549d8be292cac349fe0eff4cc5b7f8785fe1f2267154e8: Status 404 returned error can't find the container with id 32d49d35c49cfeb9fa549d8be292cac349fe0eff4cc5b7f8785fe1f2267154e8
Oct 30 12:58:54.003039 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:54.002998    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44" event={"ID":"7c46fa7a-df92-4fa8-a054-9de3c0620cca","Type":"ContainerStarted","Data":"ee104fd9fc34e49595a0cca388fe1531328c688028224b3e01cf7d35949ac9d8"}
Oct 30 12:58:54.003242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:54.003066    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44" event={"ID":"7c46fa7a-df92-4fa8-a054-9de3c0620cca","Type":"ContainerStarted","Data":"32d49d35c49cfeb9fa549d8be292cac349fe0eff4cc5b7f8785fe1f2267154e8"}
Oct 30 12:58:54.051115 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:54.051059    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44" podStartSLOduration=2.051043111 podStartE2EDuration="2.051043111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:54.046473639 +0000 UTC m=+1598.522193252" watchObservedRunningTime="2026-10-30 12:58:54.051043111 +0000 UTC m=+1598.526762753"
Oct 30 12:58:55.012309 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.012271    4993 generic.go:356] "Generic (PLEG): container finished" podID="7c46fa7a-df92-4fa8-a054-9de3c0620cca" containerID="ee104fd9fc34e49595a0cca388fe1531328c688028224b3e01cf7d35949ac9d8" exitCode=0
Oct 30 12:58:55.012493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.012358    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44" event={"ID":"7c46fa7a-df92-4fa8-a054-9de3c0620cca","Type":"ContainerDied","Data":"ee104fd9fc34e49595a0cca388fe1531328c688028224b3e01cf7d35949ac9d8"}
Oct 30 12:58:55.702397 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.702360    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"]
Oct 30 12:58:55.706700 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.706674    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.726613 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.726579    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"]
Oct 30 12:58:55.812900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.812865    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358ac826-6788-4428-b685-9e485b695339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.813074 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.812929    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmc8\" (UniqueName: \"kubernetes.io/projected/358ac826-6788-4428-b685-9e485b695339-kube-api-access-dvmc8\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.813287 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.813262    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358ac826-6788-4428-b685-9e485b695339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.914844 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.914811    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358ac826-6788-4428-b685-9e485b695339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.915013 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.914871    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358ac826-6788-4428-b685-9e485b695339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.915013 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.914911    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmc8\" (UniqueName: \"kubernetes.io/projected/358ac826-6788-4428-b685-9e485b695339-kube-api-access-dvmc8\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.915503 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.915479    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358ac826-6788-4428-b685-9e485b695339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.917409 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.917385    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358ac826-6788-4428-b685-9e485b695339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:55.939040 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:55.939012    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmc8\" (UniqueName: \"kubernetes.io/projected/358ac826-6788-4428-b685-9e485b695339-kube-api-access-dvmc8\") pod \"maas-api-revoke-sub-dbac848a-kr26t\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:56.021076 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.021053    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:56.211738 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.211712    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:56.320379 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.320290    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-openshift-ca\") pod \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") "
Oct 30 12:58:56.320379 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.320329    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c46fa7a-df92-4fa8-a054-9de3c0620cca-kube-api-access-cdg9s\" (UniqueName: \"kubernetes.io/projected/7c46fa7a-df92-4fa8-a054-9de3c0620cca-kube-api-access-cdg9s\") pod \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") "
Oct 30 12:58:56.320618 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.320492    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-xks-ca\") pod \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\" (UID: \"7c46fa7a-df92-4fa8-a054-9de3c0620cca\") "
Oct 30 12:58:56.320795 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.320763    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-openshift-ca" pod "7c46fa7a-df92-4fa8-a054-9de3c0620cca" (UID: "7c46fa7a-df92-4fa8-a054-9de3c0620cca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:56.322531 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.322508    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-xks-ca" pod "7c46fa7a-df92-4fa8-a054-9de3c0620cca" (UID: "7c46fa7a-df92-4fa8-a054-9de3c0620cca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:56.322667 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.322647    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c46fa7a-df92-4fa8-a054-9de3c0620cca-kube-api-access-cdg9s" pod "7c46fa7a-df92-4fa8-a054-9de3c0620cca" (UID: "7c46fa7a-df92-4fa8-a054-9de3c0620cca"). InnerVolumeSpecName "kube-api-access-cdg9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:56.332623 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:56.332598    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358ac826_6788_4428_b685_9e485b695339.slice/crio-a9722cc9bc4bd522d093c29af039727aed6bba40e9970c24c37b64ea8630e061 WatchSource:0}: Error finding container a9722cc9bc4bd522d093c29af039727aed6bba40e9970c24c37b64ea8630e061: Status 404 returned error can't find the container with id a9722cc9bc4bd522d093c29af039727aed6bba40e9970c24c37b64ea8630e061
Oct 30 12:58:56.333371 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.333342    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"]
Oct 30 12:58:56.421542 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.421510    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:56.421542 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.421543    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c46fa7a-df92-4fa8-a054-9de3c0620cca-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:56.421741 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.421553    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdg9s\" (UniqueName: \"kubernetes.io/projected/7c46fa7a-df92-4fa8-a054-9de3c0620cca-kube-api-access-cdg9s\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:56.987008 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.986965    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"]
Oct 30 12:58:56.987743 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.987720    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7c46fa7a-df92-4fa8-a054-9de3c0620cca" containerName="revoke-keys"
Oct 30 12:58:56.987845 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.987746    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c46fa7a-df92-4fa8-a054-9de3c0620cca" containerName="revoke-keys"
Oct 30 12:58:56.987926 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.987913    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c46fa7a-df92-4fa8-a054-9de3c0620cca" containerName="revoke-keys"
Oct 30 12:58:56.991624 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:56.991601    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.010660 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.010608    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"]
Oct 30 12:58:57.031252 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.031195    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t" event={"ID":"358ac826-6788-4428-b685-9e485b695339","Type":"ContainerStarted","Data":"00c9740e1f52ef973abea646217caf65b9960b7459f90b64239a30913219e076"}
Oct 30 12:58:57.031445 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.031268    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t" event={"ID":"358ac826-6788-4428-b685-9e485b695339","Type":"ContainerStarted","Data":"a9722cc9bc4bd522d093c29af039727aed6bba40e9970c24c37b64ea8630e061"}
Oct 30 12:58:57.033000 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.032974    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"
Oct 30 12:58:57.033117 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.032999    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44" event={"ID":"7c46fa7a-df92-4fa8-a054-9de3c0620cca","Type":"ContainerDied","Data":"32d49d35c49cfeb9fa549d8be292cac349fe0eff4cc5b7f8785fe1f2267154e8"}
Oct 30 12:58:57.033117 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.033050    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32d49d35c49cfeb9fa549d8be292cac349fe0eff4cc5b7f8785fe1f2267154e8"
Oct 30 12:58:57.130864 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.130823    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.131098 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.130933    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvlzp\" (UniqueName: \"kubernetes.io/projected/9b9c4d93-31ac-4675-9b26-8f342c320bf3-kube-api-access-nvlzp\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.131171 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.131133    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.183839 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.183614    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t" podStartSLOduration=2.183592319 podStartE2EDuration="2.183592319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:57.176292013 +0000 UTC m=+1601.652011624" watchObservedRunningTime="2026-10-30 12:58:57.183592319 +0000 UTC m=+1601.659311930"
Oct 30 12:58:57.232249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.232200    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.232433 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.232263    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvlzp\" (UniqueName: \"kubernetes.io/projected/9b9c4d93-31ac-4675-9b26-8f342c320bf3-kube-api-access-nvlzp\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.232433 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.232357    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.232900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.232873    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.234684 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.234659    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.255633 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.255602    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvlzp\" (UniqueName: \"kubernetes.io/projected/9b9c4d93-31ac-4675-9b26-8f342c320bf3-kube-api-access-nvlzp\") pod \"maas-api-revoke-sub-45be7029-76xx8\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.305790 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.305758    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:58:57.616698 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:58:57.616667    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9c4d93_31ac_4675_9b26_8f342c320bf3.slice/crio-efb7d59b3942bfd2aba1d7dd51d3612df6cf78bdfa49c825044e2433070356d4 WatchSource:0}: Error finding container efb7d59b3942bfd2aba1d7dd51d3612df6cf78bdfa49c825044e2433070356d4: Status 404 returned error can't find the container with id efb7d59b3942bfd2aba1d7dd51d3612df6cf78bdfa49c825044e2433070356d4
Oct 30 12:58:57.617180 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:57.617156    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"]
Oct 30 12:58:58.044302 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:58.044256    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8" event={"ID":"9b9c4d93-31ac-4675-9b26-8f342c320bf3","Type":"ContainerStarted","Data":"30ed7a5cd5b3d0092b9011f9db57c5d3b986f63699f0cd312468f4e96c47a255"}
Oct 30 12:58:58.044761 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:58.044330    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8" event={"ID":"9b9c4d93-31ac-4675-9b26-8f342c320bf3","Type":"ContainerStarted","Data":"efb7d59b3942bfd2aba1d7dd51d3612df6cf78bdfa49c825044e2433070356d4"}
Oct 30 12:58:58.046231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:58.046169    4993 generic.go:356] "Generic (PLEG): container finished" podID="358ac826-6788-4428-b685-9e485b695339" containerID="00c9740e1f52ef973abea646217caf65b9960b7459f90b64239a30913219e076" exitCode=0
Oct 30 12:58:58.046332 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:58.046256    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t" event={"ID":"358ac826-6788-4428-b685-9e485b695339","Type":"ContainerDied","Data":"00c9740e1f52ef973abea646217caf65b9960b7459f90b64239a30913219e076"}
Oct 30 12:58:58.095114 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:58.095052    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8" podStartSLOduration=2.095038055 podStartE2EDuration="2.095038055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:58.09196705 +0000 UTC m=+1602.567686661" watchObservedRunningTime="2026-10-30 12:58:58.095038055 +0000 UTC m=+1602.570757665"
Oct 30 12:58:59.055883 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.055843    4993 generic.go:356] "Generic (PLEG): container finished" podID="9b9c4d93-31ac-4675-9b26-8f342c320bf3" containerID="30ed7a5cd5b3d0092b9011f9db57c5d3b986f63699f0cd312468f4e96c47a255" exitCode=0
Oct 30 12:58:59.056356 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.055934    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8" event={"ID":"9b9c4d93-31ac-4675-9b26-8f342c320bf3","Type":"ContainerDied","Data":"30ed7a5cd5b3d0092b9011f9db57c5d3b986f63699f0cd312468f4e96c47a255"}
Oct 30 12:58:59.229682 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.229655    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:58:59.365753 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.365658    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358ac826-6788-4428-b685-9e485b695339-kube-api-access-dvmc8\" (UniqueName: \"kubernetes.io/projected/358ac826-6788-4428-b685-9e485b695339-kube-api-access-dvmc8\") pod \"358ac826-6788-4428-b685-9e485b695339\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") "
Oct 30 12:58:59.365932 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.365755    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358ac826-6788-4428-b685-9e485b695339-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358ac826-6788-4428-b685-9e485b695339-maas-api-xks-ca\") pod \"358ac826-6788-4428-b685-9e485b695339\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") "
Oct 30 12:58:59.365932 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.365834    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358ac826-6788-4428-b685-9e485b695339-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358ac826-6788-4428-b685-9e485b695339-maas-api-openshift-ca\") pod \"358ac826-6788-4428-b685-9e485b695339\" (UID: \"358ac826-6788-4428-b685-9e485b695339\") "
Oct 30 12:58:59.366258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.366224    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358ac826-6788-4428-b685-9e485b695339-maas-api-openshift-ca" pod "358ac826-6788-4428-b685-9e485b695339" (UID: "358ac826-6788-4428-b685-9e485b695339"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:58:59.367942 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.367921    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358ac826-6788-4428-b685-9e485b695339-maas-api-xks-ca" pod "358ac826-6788-4428-b685-9e485b695339" (UID: "358ac826-6788-4428-b685-9e485b695339"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:59.368302 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.368255    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358ac826-6788-4428-b685-9e485b695339-kube-api-access-dvmc8" pod "358ac826-6788-4428-b685-9e485b695339" (UID: "358ac826-6788-4428-b685-9e485b695339"). InnerVolumeSpecName "kube-api-access-dvmc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:59.468244 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.468167    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358ac826-6788-4428-b685-9e485b695339-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:59.468244 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.468223    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvmc8\" (UniqueName: \"kubernetes.io/projected/358ac826-6788-4428-b685-9e485b695339-kube-api-access-dvmc8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:58:59.468244 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:58:59.468242    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358ac826-6788-4428-b685-9e485b695339-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:00.070253 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.070218    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"
Oct 30 12:59:00.070253 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.070234    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t" event={"ID":"358ac826-6788-4428-b685-9e485b695339","Type":"ContainerDied","Data":"a9722cc9bc4bd522d093c29af039727aed6bba40e9970c24c37b64ea8630e061"}
Oct 30 12:59:00.070802 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.070297    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9722cc9bc4bd522d093c29af039727aed6bba40e9970c24c37b64ea8630e061"
Oct 30 12:59:00.239430 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.239390    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:59:00.381094 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.380995    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-xks-ca\") pod \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") "
Oct 30 12:59:00.381094 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.381085    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9c4d93-31ac-4675-9b26-8f342c320bf3-kube-api-access-nvlzp\" (UniqueName: \"kubernetes.io/projected/9b9c4d93-31ac-4675-9b26-8f342c320bf3-kube-api-access-nvlzp\") pod \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") "
Oct 30 12:59:00.381359 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.381342    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-openshift-ca\") pod \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\" (UID: \"9b9c4d93-31ac-4675-9b26-8f342c320bf3\") "
Oct 30 12:59:00.381872 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.381838    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-openshift-ca" pod "9b9c4d93-31ac-4675-9b26-8f342c320bf3" (UID: "9b9c4d93-31ac-4675-9b26-8f342c320bf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:00.383690 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.383654    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-xks-ca" pod "9b9c4d93-31ac-4675-9b26-8f342c320bf3" (UID: "9b9c4d93-31ac-4675-9b26-8f342c320bf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:00.383830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.383806    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9c4d93-31ac-4675-9b26-8f342c320bf3-kube-api-access-nvlzp" pod "9b9c4d93-31ac-4675-9b26-8f342c320bf3" (UID: "9b9c4d93-31ac-4675-9b26-8f342c320bf3"). InnerVolumeSpecName "kube-api-access-nvlzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:00.482818 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.482784    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:00.482818 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.482816    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvlzp\" (UniqueName: \"kubernetes.io/projected/9b9c4d93-31ac-4675-9b26-8f342c320bf3-kube-api-access-nvlzp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:00.483064 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:00.482830    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c4d93-31ac-4675-9b26-8f342c320bf3-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:01.082495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:01.082454    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8" event={"ID":"9b9c4d93-31ac-4675-9b26-8f342c320bf3","Type":"ContainerDied","Data":"efb7d59b3942bfd2aba1d7dd51d3612df6cf78bdfa49c825044e2433070356d4"}
Oct 30 12:59:01.082495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:01.082488    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"
Oct 30 12:59:01.082933 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:01.082517    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efb7d59b3942bfd2aba1d7dd51d3612df6cf78bdfa49c825044e2433070356d4"
Oct 30 12:59:01.339731 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:01.339631    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:59:01.352582 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:01.352553    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:59:02.106173 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:02.106139    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:59:06.337662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.337612    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"]
Oct 30 12:59:06.338993 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.338968    4993 cpu_manager.go:414] "containerMap: removing container" podUID="358ac826-6788-4428-b685-9e485b695339" containerName="revoke-keys"
Oct 30 12:59:06.339154 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.339140    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358ac826-6788-4428-b685-9e485b695339" containerName="revoke-keys"
Oct 30 12:59:06.339282 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.339271    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9b9c4d93-31ac-4675-9b26-8f342c320bf3" containerName="revoke-keys"
Oct 30 12:59:06.339445 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.339433    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9c4d93-31ac-4675-9b26-8f342c320bf3" containerName="revoke-keys"
Oct 30 12:59:06.339731 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.339719    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="358ac826-6788-4428-b685-9e485b695339" containerName="revoke-keys"
Oct 30 12:59:06.339818 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.339809    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9c4d93-31ac-4675-9b26-8f342c320bf3" containerName="revoke-keys"
Oct 30 12:59:06.346633 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.346593    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.386735 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.386527    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"]
Oct 30 12:59:06.398478 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.398434    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.398782 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.398765    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.398936 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.398921    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.399148 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.399135    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965c37e2-a393-4b30-aa8d-35dc060595ce-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.399309 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.399294    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hb5j\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-kube-api-access-8hb5j\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.399463 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.399449    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.399609 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.399594    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.399721 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.399707    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.399838 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.399826    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.515456 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.515410    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.517747 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.517573    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.517747 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.517632    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.518241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.518002    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.519108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.518369    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965c37e2-a393-4b30-aa8d-35dc060595ce-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.519108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.518412    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hb5j\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-kube-api-access-8hb5j\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.519108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.518998    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.519108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.518461    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.519388 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.519167    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.519388 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.519328    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.519388 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.519370    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.520102 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.519825    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.520793 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.520688    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.521118 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.520923    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965c37e2-a393-4b30-aa8d-35dc060595ce-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.629290 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.617044    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.644231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.634122    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 12:59:06.703397 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.674355    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hb5j\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-kube-api-access-8hb5j\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.709235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.704030    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:06.847359 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:06.845400    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"]
Oct 30 12:59:07.036247 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.005625    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 12:59:07.110229 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.084719    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:07.136435 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.128923    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 30 12:59:07.158030 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.157982    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.195921 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.195546    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.266059 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.265894    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.266385 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.266365    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.266596 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.266571    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.266775 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.266761    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.267053 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.267038    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.267330 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.267316    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.267540 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.267525    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.267724 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.267706    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.267877 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.267863    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t899q\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-kube-api-access-t899q\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.303602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.303479    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"]
Oct 30 12:59:07.323466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.323431    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:07.377553 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.377518    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.378164 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.378144    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.378362 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.378347    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.378491 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.378478    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.378587 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.378573    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t899q\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-kube-api-access-t899q\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.378706 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.378688    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.378833 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.378820    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.378939 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.378928    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.379040 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.379028    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.484603 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.480344    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.596860 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.579935    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.635439 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.635394    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.732317 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.652343    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.732527 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.652902    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.732613 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.653111    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.732722 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.690397    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.803235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.796405    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.855286 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.832859    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.855286 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.847698    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t899q\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-kube-api-access-t899q\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:07.869892 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:07.869821    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:08.130286 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:08.128436    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:09.096058 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.096009    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"]
Oct 30 12:59:09.144114 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.135259    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:09.155315 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.144525    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:09.155688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.155641    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:09.246954 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:59:09.246911    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:59:09.266570 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:59:09.266536    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6e4240_e2f8_4632_89b3_bfb1c1610ec5.slice/crio-0988efe6827bfd8f4d957d228a6f776735373b74a15c359bbd145faf77123f5f WatchSource:0}: Error finding container 0988efe6827bfd8f4d957d228a6f776735373b74a15c359bbd145faf77123f5f: Status 404 returned error can't find the container with id 0988efe6827bfd8f4d957d228a6f776735373b74a15c359bbd145faf77123f5f
Oct 30 12:59:09.267900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.267867    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"]
Oct 30 12:59:09.269579 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.269538    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:09.269662 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.269619    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:09.269699 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.269665    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:09.890088 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.888870    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" event={"ID":"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5","Type":"ContainerStarted","Data":"eb366daadaec2e3c89fcc0454265375a9d2df7b5065391a206e7b2aa081840d8"}
Oct 30 12:59:09.890088 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.888941    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" event={"ID":"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5","Type":"ContainerStarted","Data":"0988efe6827bfd8f4d957d228a6f776735373b74a15c359bbd145faf77123f5f"}
Oct 30 12:59:09.907018 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.906978    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" event={"ID":"965c37e2-a393-4b30-aa8d-35dc060595ce","Type":"ContainerStarted","Data":"ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6"}
Oct 30 12:59:09.907201 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:09.907043    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" event={"ID":"965c37e2-a393-4b30-aa8d-35dc060595ce","Type":"ContainerStarted","Data":"9a31b4f4c96eae78af412450028605b871b35e5b1cbf14eff425e8ad3b94520a"}
Oct 30 12:59:10.014946 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.014883    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podStartSLOduration=4.014866591 podStartE2EDuration="4.014866591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:09.963567081 +0000 UTC m=+1614.439286703" watchObservedRunningTime="2026-10-30 12:59:10.014866591 +0000 UTC m=+1614.490586202"
Oct 30 12:59:10.015814 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.015762    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podStartSLOduration=4.015753641 podStartE2EDuration="4.015753641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:10.011588174 +0000 UTC m=+1614.487307788" watchObservedRunningTime="2026-10-30 12:59:10.015753641 +0000 UTC m=+1614.491473248"
Oct 30 12:59:10.116272 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.116220    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"]
Oct 30 12:59:10.121150 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.121119    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.129896 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.129512    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:10.131341 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.131310    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:10.131469 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.131372    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 12:59:10.141778 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.141691    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:59:10.141988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.141847    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:59:10.142093 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.141870    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:59:10.142152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.141896    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:59:10.142152 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.142033    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"]
Oct 30 12:59:10.210576 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.210536    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntdkd\" (UniqueName: \"kubernetes.io/projected/fd12953c-eb02-4406-91d3-4fe27de8ecba-kube-api-access-ntdkd\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.210913 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.210892    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.211148 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.211133    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.318434 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.318369    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.318650 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.318575    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.318650 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.318643    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdkd\" (UniqueName: \"kubernetes.io/projected/fd12953c-eb02-4406-91d3-4fe27de8ecba-kube-api-access-ntdkd\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.319728 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.319703    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.321665 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.321646    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.324638 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.324620    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:10.327585 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.327560    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:59:10.327697 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.327614    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 12:59:10.345758 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.345524    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntdkd\" (UniqueName: \"kubernetes.io/projected/fd12953c-eb02-4406-91d3-4fe27de8ecba-kube-api-access-ntdkd\") pod \"maas-api-revoke-sub-1bbac5fe-vfr47\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.440268 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.440134    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:10.850017 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.849983    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"]
Oct 30 12:59:10.850146 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:59:10.850080    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd12953c_eb02_4406_91d3_4fe27de8ecba.slice/crio-14f79759d079a5981f2cb64638658c51b0b53e6766ec1e22b7d67f986ee33b99 WatchSource:0}: Error finding container 14f79759d079a5981f2cb64638658c51b0b53e6766ec1e22b7d67f986ee33b99: Status 404 returned error can't find the container with id 14f79759d079a5981f2cb64638658c51b0b53e6766ec1e22b7d67f986ee33b99
Oct 30 12:59:10.944367 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:10.944323    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47" event={"ID":"fd12953c-eb02-4406-91d3-4fe27de8ecba","Type":"ContainerStarted","Data":"14f79759d079a5981f2cb64638658c51b0b53e6766ec1e22b7d67f986ee33b99"}
Oct 30 12:59:11.130849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:11.130755    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:11.130849 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:11.130826    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 12:59:11.324523 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:11.324475    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:59:11.324675 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:11.324536    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 12:59:11.976294 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:11.962317    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47" event={"ID":"fd12953c-eb02-4406-91d3-4fe27de8ecba","Type":"ContainerStarted","Data":"8ba77f6a8f7a94d77faff3362204d3a99085ebaa9945169d3643578745fe1e17"}
Oct 30 12:59:12.015358 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:12.015296    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47" podStartSLOduration=2.015279882 podStartE2EDuration="2.015279882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:12.012981661 +0000 UTC m=+1616.488701270" watchObservedRunningTime="2026-10-30 12:59:12.015279882 +0000 UTC m=+1616.490999523"
Oct 30 12:59:12.130104 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:12.130056    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:12.130359 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:12.130120    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 12:59:12.324482 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:12.324446    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:59:12.324970 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:12.324508    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 12:59:12.998231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:12.998005    4993 generic.go:356] "Generic (PLEG): container finished" podID="fd12953c-eb02-4406-91d3-4fe27de8ecba" containerID="8ba77f6a8f7a94d77faff3362204d3a99085ebaa9945169d3643578745fe1e17" exitCode=0
Oct 30 12:59:12.998568 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:12.998312    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47" event={"ID":"fd12953c-eb02-4406-91d3-4fe27de8ecba","Type":"ContainerDied","Data":"8ba77f6a8f7a94d77faff3362204d3a99085ebaa9945169d3643578745fe1e17"}
Oct 30 12:59:13.129815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.129776    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:13.130076 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.130048    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 12:59:13.324576 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.324476    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:59:13.325163 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.325136    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 12:59:13.477232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.476366    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"]
Oct 30 12:59:13.520487 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.519610    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"]
Oct 30 12:59:13.520487 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.519736    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.633278 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.630330    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqxll\" (UniqueName: \"kubernetes.io/projected/32ee01c6-7655-49d9-bbf8-0b0762765221-kube-api-access-fqxll\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.633278 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.630411    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.633278 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.630487    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.733646 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.732987    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.733646 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.733267    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqxll\" (UniqueName: \"kubernetes.io/projected/32ee01c6-7655-49d9-bbf8-0b0762765221-kube-api-access-fqxll\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.733646 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.733349    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.736257 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.734114    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.739822 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.739786    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.790238 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.777231    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqxll\" (UniqueName: \"kubernetes.io/projected/32ee01c6-7655-49d9-bbf8-0b0762765221-kube-api-access-fqxll\") pod \"maas-api-revoke-sub-2590b741-7jfcs\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:13.838353 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:13.837000    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:14.133031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.131606    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:14.133031 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.131669    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 12:59:14.325282 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.325243    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:59:14.325797 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.325316    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 12:59:14.537349 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.537321    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"]
Oct 30 12:59:14.554227 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.548951    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:14.641900 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.641777    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-xks-ca\") pod \"fd12953c-eb02-4406-91d3-4fe27de8ecba\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") "
Oct 30 12:59:14.642224 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.642129    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-openshift-ca\") pod \"fd12953c-eb02-4406-91d3-4fe27de8ecba\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") "
Oct 30 12:59:14.643384 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.643346    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-openshift-ca" pod "fd12953c-eb02-4406-91d3-4fe27de8ecba" (UID: "fd12953c-eb02-4406-91d3-4fe27de8ecba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:14.730630 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.730595    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-xks-ca" pod "fd12953c-eb02-4406-91d3-4fe27de8ecba" (UID: "fd12953c-eb02-4406-91d3-4fe27de8ecba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:14.744393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.744360    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd12953c-eb02-4406-91d3-4fe27de8ecba-kube-api-access-ntdkd\" (UniqueName: \"kubernetes.io/projected/fd12953c-eb02-4406-91d3-4fe27de8ecba-kube-api-access-ntdkd\") pod \"fd12953c-eb02-4406-91d3-4fe27de8ecba\" (UID: \"fd12953c-eb02-4406-91d3-4fe27de8ecba\") "
Oct 30 12:59:14.745064 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.745042    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:14.745177 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.745164    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd12953c-eb02-4406-91d3-4fe27de8ecba-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:14.749441 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.749409    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd12953c-eb02-4406-91d3-4fe27de8ecba-kube-api-access-ntdkd" pod "fd12953c-eb02-4406-91d3-4fe27de8ecba" (UID: "fd12953c-eb02-4406-91d3-4fe27de8ecba"). InnerVolumeSpecName "kube-api-access-ntdkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:14.846309 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:14.846272    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntdkd\" (UniqueName: \"kubernetes.io/projected/fd12953c-eb02-4406-91d3-4fe27de8ecba-kube-api-access-ntdkd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:15.046308 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.046273    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs" event={"ID":"32ee01c6-7655-49d9-bbf8-0b0762765221","Type":"ContainerStarted","Data":"cce1c49950d250f393d2418fa5f8ac4250757e23823849db62e2f458de158f38"}
Oct 30 12:59:15.047831 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.047808    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"
Oct 30 12:59:15.047967 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.047838    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47" event={"ID":"fd12953c-eb02-4406-91d3-4fe27de8ecba","Type":"ContainerDied","Data":"14f79759d079a5981f2cb64638658c51b0b53e6766ec1e22b7d67f986ee33b99"}
Oct 30 12:59:15.047967 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.047880    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f79759d079a5981f2cb64638658c51b0b53e6766ec1e22b7d67f986ee33b99"
Oct 30 12:59:15.131007 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.130968    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:15.131186 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.131045    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 12:59:15.324479 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.324436    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:59:15.324661 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.324512    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 12:59:15.887437 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.887389    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"]
Oct 30 12:59:15.888242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.888220    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fd12953c-eb02-4406-91d3-4fe27de8ecba" containerName="revoke-keys"
Oct 30 12:59:15.888242 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.888242    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd12953c-eb02-4406-91d3-4fe27de8ecba" containerName="revoke-keys"
Oct 30 12:59:15.888484 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.888443    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd12953c-eb02-4406-91d3-4fe27de8ecba" containerName="revoke-keys"
Oct 30 12:59:15.892623 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.892597    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:15.911319 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.911294    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"]
Oct 30 12:59:15.980333 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.980281    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:15.980553 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.980482    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msn9g\" (UniqueName: \"kubernetes.io/projected/2bfc4e95-38d8-41af-8e0f-637e9a068b20-kube-api-access-msn9g\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:15.980626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:15.980558    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:16.078193 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.078091    4993 generic.go:356] "Generic (PLEG): container finished" podID="32ee01c6-7655-49d9-bbf8-0b0762765221" containerID="579e8e1e8d263bd37c4cc7818fdafafd8252c4371dd44ac90a8d4baa4f0c9885" exitCode=0
Oct 30 12:59:16.078588 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.078166    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs" event={"ID":"32ee01c6-7655-49d9-bbf8-0b0762765221","Type":"ContainerDied","Data":"579e8e1e8d263bd37c4cc7818fdafafd8252c4371dd44ac90a8d4baa4f0c9885"}
Oct 30 12:59:16.081909 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.081880    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:16.082064 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.082034    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msn9g\" (UniqueName: \"kubernetes.io/projected/2bfc4e95-38d8-41af-8e0f-637e9a068b20-kube-api-access-msn9g\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:16.082227 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.082109    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:16.083002 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.082974    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:16.085243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.085180    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:16.106615 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.106576    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msn9g\" (UniqueName: \"kubernetes.io/projected/2bfc4e95-38d8-41af-8e0f-637e9a068b20-kube-api-access-msn9g\") pod \"maas-api-revoke-sub-89a1c7b9-hvm9k\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:16.130595 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.130552    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 30 12:59:16.130810 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.130614    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 30 12:59:16.210843 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.210799    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:16.325470 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.325062    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:59:16.325470 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.325118    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 12:59:16.543840 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:16.543811    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"]
Oct 30 12:59:16.545197 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:59:16.545164    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfc4e95_38d8_41af_8e0f_637e9a068b20.slice/crio-964bfb7af988b91ce798af59265356b21393349fc041bca6a5bd45587bfe1b69 WatchSource:0}: Error finding container 964bfb7af988b91ce798af59265356b21393349fc041bca6a5bd45587bfe1b69: Status 404 returned error can't find the container with id 964bfb7af988b91ce798af59265356b21393349fc041bca6a5bd45587bfe1b69
Oct 30 12:59:17.091297 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.091259    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k" event={"ID":"2bfc4e95-38d8-41af-8e0f-637e9a068b20","Type":"ContainerStarted","Data":"1369a5fe5283f951811e352d3acc818857fce8e33e1eab81d52f4faae3ba3020"}
Oct 30 12:59:17.091801 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.091310    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k" event={"ID":"2bfc4e95-38d8-41af-8e0f-637e9a068b20","Type":"ContainerStarted","Data":"964bfb7af988b91ce798af59265356b21393349fc041bca6a5bd45587bfe1b69"}
Oct 30 12:59:17.142408 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.142321    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k" podStartSLOduration=2.142300521 podStartE2EDuration="2.142300521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:17.138844746 +0000 UTC m=+1621.614564359" watchObservedRunningTime="2026-10-30 12:59:17.142300521 +0000 UTC m=+1621.618020134"
Oct 30 12:59:17.145377 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.145175    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:17.146080 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.146055    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:17.146988 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.146963    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 12:59:17.329401 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.329322    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:17.329745 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.329716    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:17.329859 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.329814    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:17.330394 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.330376    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 12:59:17.510913 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.510881    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-xks-ca\") pod \"32ee01c6-7655-49d9-bbf8-0b0762765221\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") "
Oct 30 12:59:17.511108 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.511060    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ee01c6-7655-49d9-bbf8-0b0762765221-kube-api-access-fqxll\" (UniqueName: \"kubernetes.io/projected/32ee01c6-7655-49d9-bbf8-0b0762765221-kube-api-access-fqxll\") pod \"32ee01c6-7655-49d9-bbf8-0b0762765221\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") "
Oct 30 12:59:17.511172 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.511115    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-openshift-ca\") pod \"32ee01c6-7655-49d9-bbf8-0b0762765221\" (UID: \"32ee01c6-7655-49d9-bbf8-0b0762765221\") "
Oct 30 12:59:17.511517 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.511491    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-openshift-ca" pod "32ee01c6-7655-49d9-bbf8-0b0762765221" (UID: "32ee01c6-7655-49d9-bbf8-0b0762765221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:17.511652 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.511634    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:17.513022 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.513000    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-xks-ca" pod "32ee01c6-7655-49d9-bbf8-0b0762765221" (UID: "32ee01c6-7655-49d9-bbf8-0b0762765221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:17.513144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.513122    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ee01c6-7655-49d9-bbf8-0b0762765221-kube-api-access-fqxll" pod "32ee01c6-7655-49d9-bbf8-0b0762765221" (UID: "32ee01c6-7655-49d9-bbf8-0b0762765221"). InnerVolumeSpecName "kube-api-access-fqxll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:17.613045 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.612958    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ee01c6-7655-49d9-bbf8-0b0762765221-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:17.613045 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:17.612993    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqxll\" (UniqueName: \"kubernetes.io/projected/32ee01c6-7655-49d9-bbf8-0b0762765221-kube-api-access-fqxll\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:18.102673 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:18.102637    4993 generic.go:356] "Generic (PLEG): container finished" podID="2bfc4e95-38d8-41af-8e0f-637e9a068b20" containerID="1369a5fe5283f951811e352d3acc818857fce8e33e1eab81d52f4faae3ba3020" exitCode=0
Oct 30 12:59:18.103150 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:18.102723    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k" event={"ID":"2bfc4e95-38d8-41af-8e0f-637e9a068b20","Type":"ContainerDied","Data":"1369a5fe5283f951811e352d3acc818857fce8e33e1eab81d52f4faae3ba3020"}
Oct 30 12:59:18.104709 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:18.104670    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"
Oct 30 12:59:18.104709 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:18.104702    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs" event={"ID":"32ee01c6-7655-49d9-bbf8-0b0762765221","Type":"ContainerDied","Data":"cce1c49950d250f393d2418fa5f8ac4250757e23823849db62e2f458de158f38"}
Oct 30 12:59:18.104909 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:18.104750    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cce1c49950d250f393d2418fa5f8ac4250757e23823849db62e2f458de158f38"
Oct 30 12:59:19.285640 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.285606    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:19.332495 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.332462    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bfc4e95-38d8-41af-8e0f-637e9a068b20-kube-api-access-msn9g\" (UniqueName: \"kubernetes.io/projected/2bfc4e95-38d8-41af-8e0f-637e9a068b20-kube-api-access-msn9g\") pod \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") "
Oct 30 12:59:19.332665 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.332652    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-openshift-ca\") pod \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") "
Oct 30 12:59:19.332720 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.332711    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-xks-ca\") pod \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\" (UID: \"2bfc4e95-38d8-41af-8e0f-637e9a068b20\") "
Oct 30 12:59:19.333060 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.333023    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-openshift-ca" pod "2bfc4e95-38d8-41af-8e0f-637e9a068b20" (UID: "2bfc4e95-38d8-41af-8e0f-637e9a068b20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:19.334970 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.334934    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-xks-ca" pod "2bfc4e95-38d8-41af-8e0f-637e9a068b20" (UID: "2bfc4e95-38d8-41af-8e0f-637e9a068b20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:19.335099 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.335067    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfc4e95-38d8-41af-8e0f-637e9a068b20-kube-api-access-msn9g" pod "2bfc4e95-38d8-41af-8e0f-637e9a068b20" (UID: "2bfc4e95-38d8-41af-8e0f-637e9a068b20"). InnerVolumeSpecName "kube-api-access-msn9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:19.434215 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.434101    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msn9g\" (UniqueName: \"kubernetes.io/projected/2bfc4e95-38d8-41af-8e0f-637e9a068b20-kube-api-access-msn9g\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:19.434215 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.434136    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:19.434215 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:19.434146    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bfc4e95-38d8-41af-8e0f-637e9a068b20-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:20.121936 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.121907    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"
Oct 30 12:59:20.121936 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.121914    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k" event={"ID":"2bfc4e95-38d8-41af-8e0f-637e9a068b20","Type":"ContainerDied","Data":"964bfb7af988b91ce798af59265356b21393349fc041bca6a5bd45587bfe1b69"}
Oct 30 12:59:20.122163 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.121974    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="964bfb7af988b91ce798af59265356b21393349fc041bca6a5bd45587bfe1b69"
Oct 30 12:59:20.547304 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.547261    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"]
Oct 30 12:59:20.548647 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.548619    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2bfc4e95-38d8-41af-8e0f-637e9a068b20" containerName="revoke-keys"
Oct 30 12:59:20.548803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.548790    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bfc4e95-38d8-41af-8e0f-637e9a068b20" containerName="revoke-keys"
Oct 30 12:59:20.552739 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.548929    4993 cpu_manager.go:414] "containerMap: removing container" podUID="32ee01c6-7655-49d9-bbf8-0b0762765221" containerName="revoke-keys"
Oct 30 12:59:20.552916 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.552894    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ee01c6-7655-49d9-bbf8-0b0762765221" containerName="revoke-keys"
Oct 30 12:59:20.557225 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.553184    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ee01c6-7655-49d9-bbf8-0b0762765221" containerName="revoke-keys"
Oct 30 12:59:20.557416 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.557398    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bfc4e95-38d8-41af-8e0f-637e9a068b20" containerName="revoke-keys"
Oct 30 12:59:20.613717 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.613688    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"]
Oct 30 12:59:20.613901 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.613803    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.629938 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.629855    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:59:20.642227 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.641879    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:59:20.642227 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.642067    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:59:20.642227 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.642195    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:59:20.785758 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.785719    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.785987 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.785955    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xt4\" (UniqueName: \"kubernetes.io/projected/cb9018a3-576b-4d77-859c-4c9fc233d394-kube-api-access-r5xt4\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.786107 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.786067    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.897247 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.897128    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xt4\" (UniqueName: \"kubernetes.io/projected/cb9018a3-576b-4d77-859c-4c9fc233d394-kube-api-access-r5xt4\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.897543 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.897522    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.897713 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.897697    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.898686 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.898656    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.931086 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.931046    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xt4\" (UniqueName: \"kubernetes.io/projected/cb9018a3-576b-4d77-859c-4c9fc233d394-kube-api-access-r5xt4\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:20.942811 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:20.942775    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea617935-kcs4c\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:21.230830 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:21.230786    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:22.389020 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:22.388352    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:22.479297 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:22.479254    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:22.597368 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:22.597301    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 12:59:22.641664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:22.624656    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 12:59:22.668975 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:22.668839    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"]
Oct 30 12:59:22.669417 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:22.669378    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.176233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.152347    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.222233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.205248    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.222233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.205694    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Oct 30 12:59:23.234341 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.234295    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 12:59:23.319461 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.298077    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 12:59:23.348042 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.346844    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 12:59:23.348042 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.347307    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded"
Oct 30 12:59:23.350243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.350134    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c" event={"ID":"cb9018a3-576b-4d77-859c-4c9fc233d394","Type":"ContainerStarted","Data":"fa2038c49a9eb965b3df595872270868e9d30b6cb2178b6ac69adbc200f190fe"}
Oct 30 12:59:23.393300 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.392305    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 12:59:23.393300 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.392795    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:59:23.486573 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.483310    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.486573 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.485525    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:59:23.699465 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.656816    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.699465 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.657279    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.699465 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.657943    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:59:23.699465 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.658003    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:59:23.699465 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.675820    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.699465 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:23.676272    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:59:24.156258 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.156179    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.157699 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.157674    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:59:24.217882 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.217833    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 12:59:24.227430 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.220011    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:59:24.287250 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.276505    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.324233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.295674    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:59:24.349698 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.328968    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.621383 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.380130    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 12:59:24.648264 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.648229    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:59:24.648566 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.648547    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:59:24.662354 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.435770    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.721269 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.558334    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 12:59:24.726340 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.726303    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 12:59:24.727825 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.727794    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 12:59:24.730149 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.730109    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.730405 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.730384    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.754639 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:24.754601    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c" event={"ID":"cb9018a3-576b-4d77-859c-4c9fc233d394","Type":"ContainerStarted","Data":"25042409dbae88c007502abf189a5df013dcb98944653135abd6e6345c4ccbb9"}
Oct 30 12:59:25.166295 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.166250    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:25.255228 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.254735    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 12:59:25.290292 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.290251    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:25.317259 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.307130    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 12:59:25.317259 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.307273    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:59:25.510199 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.508950    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"]
Oct 30 12:59:25.625989 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.625595    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:25.663230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.660027    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:25.767291 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.730631    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:25.772241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.732624    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 12:59:25.774329 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:25.774295    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:26.016603 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.016557    4993 generic.go:356] "Generic (PLEG): container finished" podID="cb9018a3-576b-4d77-859c-4c9fc233d394" containerID="25042409dbae88c007502abf189a5df013dcb98944653135abd6e6345c4ccbb9" exitCode=0
Oct 30 12:59:26.029481 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.029394    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.029692 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.029674    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.029803 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.029790    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.029929 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.029916    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d67554-66c0-4518-8818-6655f02c621f-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.030047 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.030035    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.030177 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.030166    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.067018 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.066980    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d67554-66c0-4518-8818-6655f02c621f-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.067282 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.067260    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w97d\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-kube-api-access-7w97d\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.067431 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.067416    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.178426 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.178376    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 30 12:59:26.250608 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.250576    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.273731 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.271149    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c" podStartSLOduration=6.271126665 podStartE2EDuration="6.271126665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:25.336327109 +0000 UTC m=+1629.812046722" watchObservedRunningTime="2026-10-30 12:59:26.271126665 +0000 UTC m=+1630.746846277"
Oct 30 12:59:26.283737 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.283657    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"]
Oct 30 12:59:26.284125 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.284103    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c" event={"ID":"cb9018a3-576b-4d77-859c-4c9fc233d394","Type":"ContainerDied","Data":"25042409dbae88c007502abf189a5df013dcb98944653135abd6e6345c4ccbb9"}
Oct 30 12:59:26.288424 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.288403    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 12:59:26.315264 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.288481    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:26.318260 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.318225    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.324418 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.324387    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d67554-66c0-4518-8818-6655f02c621f-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.324668 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.324644    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w97d\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-kube-api-access-7w97d\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.325297 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.325274    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.325486 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.325469    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.325620 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.325603    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.325739 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.325722    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.325887 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.325869    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d67554-66c0-4518-8818-6655f02c621f-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.326027 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.326013    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.333231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.322476    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 12:59:26.345402 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.336419    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.345581 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.335407    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d67554-66c0-4518-8818-6655f02c621f-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.345675 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.335927    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.345748 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.336173    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.345827 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.334727    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.413239 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.412588    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.450231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.448848    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.458279 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.458243    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d67554-66c0-4518-8818-6655f02c621f-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.470962 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.470927    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w97d\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-kube-api-access-7w97d\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.618361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.618138    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 12:59:26.618361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.618222    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 12:59:26.618361 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.618257    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 12:59:26.637269 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.633702    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 12:59:26.645831 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.642828    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"]
Oct 30 12:59:26.646934 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.646901    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:26.650196 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.649679    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.659365 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.659332    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:26.712392 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.712198    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 12:59:26.733106 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.732753    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97edabe4-74b8-4125-a2be-642e0470eba5-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.733389 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.733353    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.733628 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.733597    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.733948 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.733927    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.734123 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.734104    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97edabe4-74b8-4125-a2be-642e0470eba5-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.734378 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.734357    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.734498 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.734482    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.734645 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.734627    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v5rz\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-kube-api-access-4v5rz\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.734815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.734797    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.754231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.752104    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:59:26.848019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.847968    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.849717 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.849693    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97edabe4-74b8-4125-a2be-642e0470eba5-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.865963 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.865841    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.866330 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.866309    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.866462 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.866448    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5rz\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-kube-api-access-4v5rz\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.866815 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.866800    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.867126 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.867085    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97edabe4-74b8-4125-a2be-642e0470eba5-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.867247 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.867142    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.867247 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.867182    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.868312 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.853495    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.868312 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.854147    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97edabe4-74b8-4125-a2be-642e0470eba5-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.868312 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.867913    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.870508 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.870440    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.870720 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.870629    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.871154 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.871139    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"]
Oct 30 12:59:26.901493 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.894315    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.916727 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.916691    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97edabe4-74b8-4125-a2be-642e0470eba5-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.995673 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.994062    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5rz\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-kube-api-access-4v5rz\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:26.995673 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:26.995173    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:27.153529 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.153448    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:27.280824 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:59:27.280787    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d67554_66c0_4518_8818_6655f02c621f.slice/crio-5dd04abe7286e5fdef8338c0a11740980db47554ba2910ed3861c5980d237992 WatchSource:0}: Error finding container 5dd04abe7286e5fdef8338c0a11740980db47554ba2910ed3861c5980d237992: Status 404 returned error can't find the container with id 5dd04abe7286e5fdef8338c0a11740980db47554ba2910ed3861c5980d237992
Oct 30 12:59:27.296222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.294490    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:27.296222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.294591    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:27.296222 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.294635    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:27.314523 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.307014    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"]
Oct 30 12:59:27.344734 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.344702    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:27.392009 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.391975    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-openshift-ca\") pod \"cb9018a3-576b-4d77-859c-4c9fc233d394\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") "
Oct 30 12:59:27.392150 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.392115    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9018a3-576b-4d77-859c-4c9fc233d394-kube-api-access-r5xt4\" (UniqueName: \"kubernetes.io/projected/cb9018a3-576b-4d77-859c-4c9fc233d394-kube-api-access-r5xt4\") pod \"cb9018a3-576b-4d77-859c-4c9fc233d394\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") "
Oct 30 12:59:27.392376 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.392354    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-xks-ca\") pod \"cb9018a3-576b-4d77-859c-4c9fc233d394\" (UID: \"cb9018a3-576b-4d77-859c-4c9fc233d394\") "
Oct 30 12:59:27.401401 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.399460    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-openshift-ca" pod "cb9018a3-576b-4d77-859c-4c9fc233d394" (UID: "cb9018a3-576b-4d77-859c-4c9fc233d394"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:27.409451 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.409378    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-xks-ca" pod "cb9018a3-576b-4d77-859c-4c9fc233d394" (UID: "cb9018a3-576b-4d77-859c-4c9fc233d394"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:27.409451 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.409404    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9018a3-576b-4d77-859c-4c9fc233d394-kube-api-access-r5xt4" pod "cb9018a3-576b-4d77-859c-4c9fc233d394" (UID: "cb9018a3-576b-4d77-859c-4c9fc233d394"). InnerVolumeSpecName "kube-api-access-r5xt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:27.501630 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.501242    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:27.501630 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.501279    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9018a3-576b-4d77-859c-4c9fc233d394-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:27.501630 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.501295    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5xt4\" (UniqueName: \"kubernetes.io/projected/cb9018a3-576b-4d77-859c-4c9fc233d394-kube-api-access-r5xt4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:27.527465 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.527258    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"]
Oct 30 12:59:27.562466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.561121    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:27.562466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.561235    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:27.562466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:27.561278    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:28.108664 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.108625    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c" event={"ID":"cb9018a3-576b-4d77-859c-4c9fc233d394","Type":"ContainerDied","Data":"fa2038c49a9eb965b3df595872270868e9d30b6cb2178b6ac69adbc200f190fe"}
Oct 30 12:59:28.109156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.108680    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa2038c49a9eb965b3df595872270868e9d30b6cb2178b6ac69adbc200f190fe"
Oct 30 12:59:28.109156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.108735    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"
Oct 30 12:59:28.111148 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.111117    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" event={"ID":"45d67554-66c0-4518-8818-6655f02c621f","Type":"ContainerStarted","Data":"96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f"}
Oct 30 12:59:28.111366 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.111346    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" event={"ID":"45d67554-66c0-4518-8818-6655f02c621f","Type":"ContainerStarted","Data":"5dd04abe7286e5fdef8338c0a11740980db47554ba2910ed3861c5980d237992"}
Oct 30 12:59:28.126628 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.126585    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" event={"ID":"97edabe4-74b8-4125-a2be-642e0470eba5","Type":"ContainerStarted","Data":"b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267"}
Oct 30 12:59:28.126904 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.126882    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" event={"ID":"97edabe4-74b8-4125-a2be-642e0470eba5","Type":"ContainerStarted","Data":"9a16e9de38e65d03680f5eb7816e32588d07e6963be831e28ce441a2c7477a76"}
Oct 30 12:59:28.165877 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.158082    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podStartSLOduration=4.158064286 podStartE2EDuration="4.158064286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:28.156736135 +0000 UTC m=+1632.632455830" watchObservedRunningTime="2026-10-30 12:59:28.158064286 +0000 UTC m=+1632.633783897"
Oct 30 12:59:28.165877 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.164545    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:28.166578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.166266    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 12:59:28.166578 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.166324    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 12:59:28.226675 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.226623    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podStartSLOduration=3.226608188 podStartE2EDuration="3.226608188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:28.222777791 +0000 UTC m=+1632.698497401" watchObservedRunningTime="2026-10-30 12:59:28.226608188 +0000 UTC m=+1632.702327867"
Oct 30 12:59:28.666956 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.666239    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:28.687231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.685047    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 12:59:28.687231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:28.685125    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 12:59:29.154602 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:29.154563    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 12:59:29.155158 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:29.155128    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 12:59:29.671230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:29.665646    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 12:59:29.671230 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:29.665729    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 12:59:30.156233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:30.155859    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 12:59:30.156233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:30.155921    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 12:59:30.661748 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:30.661707    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 12:59:30.662719 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:30.662689    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 12:59:31.173232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:31.172707    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 12:59:31.173232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:31.172774    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 12:59:31.662232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:31.661672    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 12:59:31.662232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:31.661745    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 12:59:32.156599 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:32.156561    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 12:59:32.156883 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:32.156855    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 12:59:32.668231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:32.661593    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 12:59:32.668231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:32.661660    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 12:59:33.161712 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:33.161673    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 12:59:33.162035 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:33.161991    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 12:59:33.670116 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:33.670060    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 12:59:33.670793 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:33.670742    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 12:59:34.154762 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:34.154723    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 12:59:34.154945 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:34.154798    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 12:59:34.689423 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:34.689377    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 12:59:34.689954 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:34.689445    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 12:59:35.156000 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.154660    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 12:59:35.156000 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.154726    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 12:59:35.665743 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.665706    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:35.666097 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.666074    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:35.666965 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.666943    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 12:59:35.719164 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.719131    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"]
Oct 30 12:59:35.719715 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.719696    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cb9018a3-576b-4d77-859c-4c9fc233d394" containerName="revoke-keys"
Oct 30 12:59:35.719715 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.719714    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9018a3-576b-4d77-859c-4c9fc233d394" containerName="revoke-keys"
Oct 30 12:59:35.719842 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.719819    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9018a3-576b-4d77-859c-4c9fc233d394" containerName="revoke-keys"
Oct 30 12:59:35.723388 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.723357    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.743225 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.739957    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:59:35.743225 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.742653    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:59:35.743606 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.743577    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:59:35.744022 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.743854    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:59:35.768535 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.768501    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"]
Oct 30 12:59:35.770413 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.770375    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.770580 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.770533    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7tbr\" (UniqueName: \"kubernetes.io/projected/d5628793-8ea3-4803-9c30-c25e98c635e6-kube-api-access-w7tbr\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.770688 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.770669    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.879144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.872119    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.879144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.875552    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.879144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.875670    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7tbr\" (UniqueName: \"kubernetes.io/projected/d5628793-8ea3-4803-9c30-c25e98c635e6-kube-api-access-w7tbr\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.879144 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.876953    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.886037 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.885995    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:35.916407 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:35.916326    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7tbr\" (UniqueName: \"kubernetes.io/projected/d5628793-8ea3-4803-9c30-c25e98c635e6-kube-api-access-w7tbr\") pod \"maas-api-revoke-sub-eedec044-xrwsq\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:36.045019 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:36.044983    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:36.159872 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:36.159842    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:36.160466 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:36.160405    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:36.161634 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:36.161607    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 12:59:36.360131 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:59:36.360095    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5628793_8ea3_4803_9c30_c25e98c635e6.slice/crio-e62f1e72ae1f82e2e7d4140783792cc5a7172a5bafa4277cbdb08961987d119b WatchSource:0}: Error finding container e62f1e72ae1f82e2e7d4140783792cc5a7172a5bafa4277cbdb08961987d119b: Status 404 returned error can't find the container with id e62f1e72ae1f82e2e7d4140783792cc5a7172a5bafa4277cbdb08961987d119b
Oct 30 12:59:36.360555 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:36.360529    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"]
Oct 30 12:59:37.371041 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:37.370996    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq" event={"ID":"d5628793-8ea3-4803-9c30-c25e98c635e6","Type":"ContainerStarted","Data":"40f01d08517b43620d2a2ea8bf366344c27cd58e257e51324b1466cf6447f827"}
Oct 30 12:59:37.371403 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:37.371074    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq" event={"ID":"d5628793-8ea3-4803-9c30-c25e98c635e6","Type":"ContainerStarted","Data":"e62f1e72ae1f82e2e7d4140783792cc5a7172a5bafa4277cbdb08961987d119b"}
Oct 30 12:59:38.381845 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.381808    4993 generic.go:356] "Generic (PLEG): container finished" podID="d5628793-8ea3-4803-9c30-c25e98c635e6" containerID="40f01d08517b43620d2a2ea8bf366344c27cd58e257e51324b1466cf6447f827" exitCode=0
Oct 30 12:59:38.382314 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.381891    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq" event={"ID":"d5628793-8ea3-4803-9c30-c25e98c635e6","Type":"ContainerDied","Data":"40f01d08517b43620d2a2ea8bf366344c27cd58e257e51324b1466cf6447f827"}
Oct 30 12:59:38.541759 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.541734    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:38.572332 ip-10-0-135-24 kubenswrapper[4993]: E1030 12:59:38.572292    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 12:59:38.613167 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.613126    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5628793-8ea3-4803-9c30-c25e98c635e6-kube-api-access-w7tbr\" (UniqueName: \"kubernetes.io/projected/d5628793-8ea3-4803-9c30-c25e98c635e6-kube-api-access-w7tbr\") pod \"d5628793-8ea3-4803-9c30-c25e98c635e6\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") "
Oct 30 12:59:38.613377 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.613281    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-xks-ca\") pod \"d5628793-8ea3-4803-9c30-c25e98c635e6\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") "
Oct 30 12:59:38.613377 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.613354    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-openshift-ca\") pod \"d5628793-8ea3-4803-9c30-c25e98c635e6\" (UID: \"d5628793-8ea3-4803-9c30-c25e98c635e6\") "
Oct 30 12:59:38.613784 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.613749    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-openshift-ca" pod "d5628793-8ea3-4803-9c30-c25e98c635e6" (UID: "d5628793-8ea3-4803-9c30-c25e98c635e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:38.613952 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.613895    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:38.615979 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.615945    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5628793-8ea3-4803-9c30-c25e98c635e6-kube-api-access-w7tbr" pod "d5628793-8ea3-4803-9c30-c25e98c635e6" (UID: "d5628793-8ea3-4803-9c30-c25e98c635e6"). InnerVolumeSpecName "kube-api-access-w7tbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:38.616424 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.616397    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-xks-ca" pod "d5628793-8ea3-4803-9c30-c25e98c635e6" (UID: "d5628793-8ea3-4803-9c30-c25e98c635e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:38.730625 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.730583    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7tbr\" (UniqueName: \"kubernetes.io/projected/d5628793-8ea3-4803-9c30-c25e98c635e6-kube-api-access-w7tbr\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:38.733576 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:38.733552    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5628793-8ea3-4803-9c30-c25e98c635e6-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:39.392756 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:39.392726    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"
Oct 30 12:59:39.393198 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:39.392758    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq" event={"ID":"d5628793-8ea3-4803-9c30-c25e98c635e6","Type":"ContainerDied","Data":"e62f1e72ae1f82e2e7d4140783792cc5a7172a5bafa4277cbdb08961987d119b"}
Oct 30 12:59:39.393198 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:39.392815    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e62f1e72ae1f82e2e7d4140783792cc5a7172a5bafa4277cbdb08961987d119b"
Oct 30 12:59:40.925036 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.924993    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"]
Oct 30 12:59:40.925887 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.925865    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d5628793-8ea3-4803-9c30-c25e98c635e6" containerName="revoke-keys"
Oct 30 12:59:40.925986 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.925889    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5628793-8ea3-4803-9c30-c25e98c635e6" containerName="revoke-keys"
Oct 30 12:59:40.926047 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.926036    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5628793-8ea3-4803-9c30-c25e98c635e6" containerName="revoke-keys"
Oct 30 12:59:40.930567 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.930545    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.954778 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.954747    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"]
Oct 30 12:59:40.992415 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.992330    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.992415 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.992410    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.992685 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.992445    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.992685 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.992487    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.992685 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.992548    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.992685 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.992575    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.993410 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.993381    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.993539 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.993456    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4b2n\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-kube-api-access-t4b2n\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:40.993539 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:40.993510    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095088 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095051    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095293 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095105    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095293 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095238    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095293 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095284    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095450 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095343    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095450 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095373    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095554 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095503    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095634 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095609    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4b2n\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-kube-api-access-t4b2n\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095692 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095671    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095729 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095616    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095882 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095863    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.095947 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.095912    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.096049 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.096025    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.096301 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.096279    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.097411 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.097394    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.098036 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.098019    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.118536 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.118502    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.118656 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.118599    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4b2n\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-kube-api-access-t4b2n\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.246460 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.246416    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:41.571400 ip-10-0-135-24 kubenswrapper[4993]: W1030 12:59:41.571362    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode197a8f0_a758_4a8f_a88c_bbc09e1ea50b.slice/crio-bcdcbebe4d208a8647a7f43ef3035130f2d8036370d9ce068fad80710af0e183 WatchSource:0}: Error finding container bcdcbebe4d208a8647a7f43ef3035130f2d8036370d9ce068fad80710af0e183: Status 404 returned error can't find the container with id bcdcbebe4d208a8647a7f43ef3035130f2d8036370d9ce068fad80710af0e183
Oct 30 12:59:41.572050 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.572021    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"]
Oct 30 12:59:41.574841 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.574787    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:41.574939 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.574905    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:41.574980 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:41.574945    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:42.427657 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:42.427620    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" event={"ID":"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b","Type":"ContainerStarted","Data":"174bda2ab6ac29a784a66e8657d23d6b39173c73a8cf3993c1c51c488a2a1ef2"}
Oct 30 12:59:42.428065 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:42.427708    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" event={"ID":"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b","Type":"ContainerStarted","Data":"bcdcbebe4d208a8647a7f43ef3035130f2d8036370d9ce068fad80710af0e183"}
Oct 30 12:59:42.493773 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:42.493695    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podStartSLOduration=2.493675236 podStartE2EDuration="2.493675236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:42.490899712 +0000 UTC m=+1646.966619321" watchObservedRunningTime="2026-10-30 12:59:42.493675236 +0000 UTC m=+1646.969394846"
Oct 30 12:59:43.246595 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:43.246556    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:43.248503 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:43.248474    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 12:59:43.248643 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:43.248563    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 12:59:44.247261 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:44.247222    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 12:59:44.247800 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:44.247291    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 12:59:45.247146 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:45.247103    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 12:59:45.247359 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:45.247167    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 12:59:46.091661 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.091623    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"]
Oct 30 12:59:46.112673 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.112461    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.113335 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.113303    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"]
Oct 30 12:59:46.130004 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.129980    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 12:59:46.130156 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.130143    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ljml6\""
Oct 30 12:59:46.130218 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.130186    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 12:59:46.130260 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.130249    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 12:59:46.142063 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.142033    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kd8t\" (UniqueName: \"kubernetes.io/projected/c2c5cff9-185e-482f-8a61-b2ec954a3c14-kube-api-access-7kd8t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.142235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.142073    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c5cff9-185e-482f-8a61-b2ec954a3c14-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.142235 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.142093    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.142428 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.142397    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.142481 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.142443    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.142598 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.142578    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.142658 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.142618    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.244141 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.244100    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.244141 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.244144    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.244423 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.244263    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.244423 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.244294    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.244533 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.244422    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kd8t\" (UniqueName: \"kubernetes.io/projected/c2c5cff9-185e-482f-8a61-b2ec954a3c14-kube-api-access-7kd8t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.244533 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.244457    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c5cff9-185e-482f-8a61-b2ec954a3c14-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.244533 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.244486    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.245218 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.245169    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.246799 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.246776    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.246916 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.246827    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 12:59:46.246916 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.246892    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 12:59:46.247033 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.247008    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.247198 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.247181    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c5cff9-185e-482f-8a61-b2ec954a3c14-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.247274 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.247244    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.274935 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.274905    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kd8t\" (UniqueName: \"kubernetes.io/projected/c2c5cff9-185e-482f-8a61-b2ec954a3c14-kube-api-access-7kd8t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.289231 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.286772    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c5cff9-185e-482f-8a61-b2ec954a3c14-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mnrt8\" (UID: \"c2c5cff9-185e-482f-8a61-b2ec954a3c14\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.429995 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.429903    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:46.865243 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.861985    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"]
Oct 30 12:59:46.891674 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:46.884970    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:59:47.247307 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:47.247270    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 12:59:47.247489 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:47.247327    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 12:59:47.602248 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:47.602125    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8" event={"ID":"c2c5cff9-185e-482f-8a61-b2ec954a3c14","Type":"ContainerStarted","Data":"590b5b70e291ac3da341aab4c599d3bc6f9fa177ff0c888d56e86bf3a7454b7b"}
Oct 30 12:59:48.108873 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.108833    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"]
Oct 30 12:59:48.113575 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.113545    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.133143 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.129875    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"]
Oct 30 12:59:48.134638 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.134595    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:59:48.134787 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.134652    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 12:59:48.134787 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.134669    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:59:48.134787 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.134765    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:59:48.206735 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.206689    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888693a-b071-45e0-af82-421962cd2416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.206930 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.206847    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888693a-b071-45e0-af82-421962cd2416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.207046 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.207027    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfsj\" (UniqueName: \"kubernetes.io/projected/8888693a-b071-45e0-af82-421962cd2416-kube-api-access-2kfsj\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.247420 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.247371    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 12:59:48.247626 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.247449    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 12:59:48.308540 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.308497    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888693a-b071-45e0-af82-421962cd2416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.308728 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.308680    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfsj\" (UniqueName: \"kubernetes.io/projected/8888693a-b071-45e0-af82-421962cd2416-kube-api-access-2kfsj\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.308784 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.308744    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888693a-b071-45e0-af82-421962cd2416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.309241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.309191    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888693a-b071-45e0-af82-421962cd2416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.311718 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.311682    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888693a-b071-45e0-af82-421962cd2416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.333862 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.333822    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfsj\" (UniqueName: \"kubernetes.io/projected/8888693a-b071-45e0-af82-421962cd2416-kube-api-access-2kfsj\") pod \"maas-api-revoke-sub-24a6d79d-c7h5b\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.443328 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.443236    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:48.783712 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:48.783607    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"]
Oct 30 12:59:49.473698 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:49.472528    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:49.473698 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:49.473556    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:49.475364 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:49.475253    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 12:59:49.683925 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:49.683134    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b" event={"ID":"8888693a-b071-45e0-af82-421962cd2416","Type":"ContainerStarted","Data":"cf8ba0a37d1b8dab30b1a8f75f20f6de6758fe58fe860a8d8bd202977fdbb470"}
Oct 30 12:59:49.683925 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:49.683236    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b" event={"ID":"8888693a-b071-45e0-af82-421962cd2416","Type":"ContainerStarted","Data":"ea2523c897cf7f6ebf859adb23c151d7f5b1e132dc1808b2e5a795a679db3422"}
Oct 30 12:59:49.737313 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:49.737249    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b" podStartSLOduration=1.737229337 podStartE2EDuration="1.737229337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:49.730584795 +0000 UTC m=+1654.206304412" watchObservedRunningTime="2026-10-30 12:59:49.737229337 +0000 UTC m=+1654.212948947"
Oct 30 12:59:50.540486 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:50.540440    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:50.540858 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:50.540554    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:50.723268 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:50.722994    4993 generic.go:356] "Generic (PLEG): container finished" podID="8888693a-b071-45e0-af82-421962cd2416" containerID="cf8ba0a37d1b8dab30b1a8f75f20f6de6758fe58fe860a8d8bd202977fdbb470" exitCode=0
Oct 30 12:59:50.723460 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:50.723426    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b" event={"ID":"8888693a-b071-45e0-af82-421962cd2416","Type":"ContainerDied","Data":"cf8ba0a37d1b8dab30b1a8f75f20f6de6758fe58fe860a8d8bd202977fdbb470"}
Oct 30 12:59:51.848241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:51.846941    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8" event={"ID":"c2c5cff9-185e-482f-8a61-b2ec954a3c14","Type":"ContainerStarted","Data":"3c7a26c0f8d575c2656a5e23e1dc2aed4c0594da6f1c9ca839acc5b6868a892c"}
Oct 30 12:59:51.848241 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:51.847013    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:51.852488 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:51.852452    4993 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mnrt8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:59:51.852758 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:51.852728    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8" podUID="c2c5cff9-185e-482f-8a61-b2ec954a3c14" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:59:51.920239 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:51.919882    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8" podStartSLOduration=2.266324184 podStartE2EDuration="5.919859687s" totalImagesPullingTime="3.653535503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:51.916789255 +0000 UTC m=+1656.392508877" watchObservedRunningTime="2026-10-30 12:59:51.919859687 +0000 UTC m=+1656.395579297"
Oct 30 12:59:52.679477 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:52.637330    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:52.747310 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:52.730313    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:52.747310 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:52.735312    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 12:59:52.774249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:52.766328    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:52.774249 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:52.766426    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 12:59:52.993721 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:52.993693    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mnrt8"
Oct 30 12:59:53.348637 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.216904    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.348637 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.288311    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.348637 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.290088    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.348637 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.307440    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.348637 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.343874    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.348637 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.345182    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:53.505233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.490201    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8888693a-b071-45e0-af82-421962cd2416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888693a-b071-45e0-af82-421962cd2416-maas-api-openshift-ca\") pod \"8888693a-b071-45e0-af82-421962cd2416\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") "
Oct 30 12:59:53.505233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.490524    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8888693a-b071-45e0-af82-421962cd2416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888693a-b071-45e0-af82-421962cd2416-maas-api-xks-ca\") pod \"8888693a-b071-45e0-af82-421962cd2416\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") "
Oct 30 12:59:53.505233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.491011    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8888693a-b071-45e0-af82-421962cd2416-kube-api-access-2kfsj\" (UniqueName: \"kubernetes.io/projected/8888693a-b071-45e0-af82-421962cd2416-kube-api-access-2kfsj\") pod \"8888693a-b071-45e0-af82-421962cd2416\" (UID: \"8888693a-b071-45e0-af82-421962cd2416\") "
Oct 30 12:59:53.505233 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.500509    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8888693a-b071-45e0-af82-421962cd2416-maas-api-openshift-ca" pod "8888693a-b071-45e0-af82-421962cd2416" (UID: "8888693a-b071-45e0-af82-421962cd2416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:59:53.516236 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.515483    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888693a-b071-45e0-af82-421962cd2416-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:53.595462 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.558585    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8888693a-b071-45e0-af82-421962cd2416-kube-api-access-2kfsj" pod "8888693a-b071-45e0-af82-421962cd2416" (UID: "8888693a-b071-45e0-af82-421962cd2416"). InnerVolumeSpecName "kube-api-access-2kfsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:59:53.595462 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.566916    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8888693a-b071-45e0-af82-421962cd2416-maas-api-xks-ca" pod "8888693a-b071-45e0-af82-421962cd2416" (UID: "8888693a-b071-45e0-af82-421962cd2416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:59:53.712296 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.693385    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888693a-b071-45e0-af82-421962cd2416-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:53.712296 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.693421    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kfsj\" (UniqueName: \"kubernetes.io/projected/8888693a-b071-45e0-af82-421962cd2416-kube-api-access-2kfsj\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 12:59:53.712296 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.706768    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.743117 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.742314    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.867792 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:53.867757    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:59:54.308793 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.308746    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 12:59:54.324232 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.309806    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 12:59:54.451090 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.451059    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 12:59:54.642393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.571118    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"
Oct 30 12:59:54.642393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.611249    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:59:54.642393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.611310    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:54.642393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.612692    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b" event={"ID":"8888693a-b071-45e0-af82-421962cd2416","Type":"ContainerDied","Data":"ea2523c897cf7f6ebf859adb23c151d7f5b1e132dc1808b2e5a795a679db3422"}
Oct 30 12:59:54.642393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.612758    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea2523c897cf7f6ebf859adb23c151d7f5b1e132dc1808b2e5a795a679db3422"
Oct 30 12:59:54.642393 ip-10-0-135-24 kubenswrapper[4993]: I1030 12:59:54.612814    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:00:00.324255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.324219    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"]
Oct 30 13:00:00.328880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.325567    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8888693a-b071-45e0-af82-421962cd2416" containerName="revoke-keys"
Oct 30 13:00:00.329447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.329425    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8888693a-b071-45e0-af82-421962cd2416" containerName="revoke-keys"
Oct 30 13:00:00.329863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.329845    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8888693a-b071-45e0-af82-421962cd2416" containerName="revoke-keys"
Oct 30 13:00:00.360261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.345884    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:00.360474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.360264    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"]
Oct 30 13:00:00.365614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.365491    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"]
Oct 30 13:00:00.387262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.387039    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:00.391963 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.389442    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:00.411979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.399393    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8jg\" (UniqueName: \"kubernetes.io/projected/6f8d98a9-d3fd-43d9-8334-f630bd89606c-kube-api-access-2m8jg\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l\" (UID: \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:00.411979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.399690    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d98a9-d3fd-43d9-8334-f630bd89606c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l\" (UID: \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:00.411979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.400720    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:00:00.411979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.400968    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:00:00.411979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.410568    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"]
Oct 30 13:00:00.424403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.424052    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:00:00.431234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.428858    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:00.435973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.435929    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"]
Oct 30 13:00:00.510514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.506818    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf\" (UID: \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:00.510514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.506881    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcb7\" (UniqueName: \"kubernetes.io/projected/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-kube-api-access-zqcb7\") pod \"maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf\" (UID: \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:00.510514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.507089    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8jg\" (UniqueName: \"kubernetes.io/projected/6f8d98a9-d3fd-43d9-8334-f630bd89606c-kube-api-access-2m8jg\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l\" (UID: \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:00.510514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.507171    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d98a9-d3fd-43d9-8334-f630bd89606c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l\" (UID: \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:00.517825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.513639    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d98a9-d3fd-43d9-8334-f630bd89606c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l\" (UID: \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:00.524103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.524067    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"]
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.621247    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"]
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.623712    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf\" (UID: \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.623767    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcb7\" (UniqueName: \"kubernetes.io/projected/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-kube-api-access-zqcb7\") pod \"maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf\" (UID: \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.623930    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439c6141-1827-4c42-b736-49e291f4672c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn\" (UID: \"439c6141-1827-4c42-b736-49e291f4672c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.623972    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94059e04-7219-4019-bd04-2c98bc58d2fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2\" (UID: \"94059e04-7219-4019-bd04-2c98bc58d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.624025    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plch\" (UniqueName: \"kubernetes.io/projected/439c6141-1827-4c42-b736-49e291f4672c-kube-api-access-5plch\") pod \"maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn\" (UID: \"439c6141-1827-4c42-b736-49e291f4672c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.624057    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68qd\" (UniqueName: \"kubernetes.io/projected/94059e04-7219-4019-bd04-2c98bc58d2fb-kube-api-access-b68qd\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2\" (UID: \"94059e04-7219-4019-bd04-2c98bc58d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.630357    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"]
Oct 30 13:00:00.645382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.640701    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"]
Oct 30 13:00:00.653827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.653741    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf\" (UID: \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:00.657632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.657545    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8jg\" (UniqueName: \"kubernetes.io/projected/6f8d98a9-d3fd-43d9-8334-f630bd89606c-kube-api-access-2m8jg\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l\" (UID: \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:00.669240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.668064    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:00.705404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.705194    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcb7\" (UniqueName: \"kubernetes.io/projected/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-kube-api-access-zqcb7\") pod \"maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf\" (UID: \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:00.706472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.706275    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"]
Oct 30 13:00:00.724825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.724787    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plch\" (UniqueName: \"kubernetes.io/projected/439c6141-1827-4c42-b736-49e291f4672c-kube-api-access-5plch\") pod \"maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn\" (UID: \"439c6141-1827-4c42-b736-49e291f4672c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:00.725018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.724837    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68qd\" (UniqueName: \"kubernetes.io/projected/94059e04-7219-4019-bd04-2c98bc58d2fb-kube-api-access-b68qd\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2\" (UID: \"94059e04-7219-4019-bd04-2c98bc58d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:00.726436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.725058    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439c6141-1827-4c42-b736-49e291f4672c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn\" (UID: \"439c6141-1827-4c42-b736-49e291f4672c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:00.726436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.725090    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68d1f1-747c-4b69-a51b-6f29cfc93212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9\" (UID: \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:00.726436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.725119    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94059e04-7219-4019-bd04-2c98bc58d2fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2\" (UID: \"94059e04-7219-4019-bd04-2c98bc58d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:00.726436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.725155    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94bb\" (UniqueName: \"kubernetes.io/projected/8a68d1f1-747c-4b69-a51b-6f29cfc93212-kube-api-access-t94bb\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9\" (UID: \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:00.726436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.725962    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94059e04-7219-4019-bd04-2c98bc58d2fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2\" (UID: \"94059e04-7219-4019-bd04-2c98bc58d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:00.727075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.727043    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439c6141-1827-4c42-b736-49e291f4672c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn\" (UID: \"439c6141-1827-4c42-b736-49e291f4672c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:00.736988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.734953    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:00.748951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.748914    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plch\" (UniqueName: \"kubernetes.io/projected/439c6141-1827-4c42-b736-49e291f4672c-kube-api-access-5plch\") pod \"maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn\" (UID: \"439c6141-1827-4c42-b736-49e291f4672c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:00.750662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.750638    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68qd\" (UniqueName: \"kubernetes.io/projected/94059e04-7219-4019-bd04-2c98bc58d2fb-kube-api-access-b68qd\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2\" (UID: \"94059e04-7219-4019-bd04-2c98bc58d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:00.832358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.832313    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68d1f1-747c-4b69-a51b-6f29cfc93212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9\" (UID: \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:00.832556 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.832389    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94bb\" (UniqueName: \"kubernetes.io/projected/8a68d1f1-747c-4b69-a51b-6f29cfc93212-kube-api-access-t94bb\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9\" (UID: \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:00.833295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.833259    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68d1f1-747c-4b69-a51b-6f29cfc93212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9\" (UID: \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:00.857870 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.857803    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94bb\" (UniqueName: \"kubernetes.io/projected/8a68d1f1-747c-4b69-a51b-6f29cfc93212-kube-api-access-t94bb\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9\" (UID: \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:00.907975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.907491    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:00.907975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.907921    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:00.973226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:00.971774    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:01.035691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.032688    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:01.126531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.126493    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"]
Oct 30 13:00:01.128792 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:00:01.128759    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8d98a9_d3fd_43d9_8334_f630bd89606c.slice/crio-41231c9b67f2c6ada7b67c144605b67bf5badbb4328c244fd22535f8387245e2 WatchSource:0}: Error finding container 41231c9b67f2c6ada7b67c144605b67bf5badbb4328c244fd22535f8387245e2: Status 404 returned error can't find the container with id 41231c9b67f2c6ada7b67c144605b67bf5badbb4328c244fd22535f8387245e2
Oct 30 13:00:01.250392 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:00:01.250352    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439c6141_1827_4c42_b736_49e291f4672c.slice/crio-cb8233717e74c499019b58990018d005e555ee22ba2c189cdee3ae7fe8874393 WatchSource:0}: Error finding container cb8233717e74c499019b58990018d005e555ee22ba2c189cdee3ae7fe8874393: Status 404 returned error can't find the container with id cb8233717e74c499019b58990018d005e555ee22ba2c189cdee3ae7fe8874393
Oct 30 13:00:01.251000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.250977    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"]
Oct 30 13:00:01.267146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.267108    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"]
Oct 30 13:00:01.272263 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:00:01.272187    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fed7f4c_7755_49ef_ab10_0ee91cfd903d.slice/crio-0e0002a674f10cc1d6815fbee837a0c428c72d50418abac0455de5525eefb2cd WatchSource:0}: Error finding container 0e0002a674f10cc1d6815fbee837a0c428c72d50418abac0455de5525eefb2cd: Status 404 returned error can't find the container with id 0e0002a674f10cc1d6815fbee837a0c428c72d50418abac0455de5525eefb2cd
Oct 30 13:00:01.376690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.376643    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"]
Oct 30 13:00:01.377098 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:00:01.376900    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94059e04_7219_4019_bd04_2c98bc58d2fb.slice/crio-634cb55605d82d7b822507335571ed3815d125e7ba432588d02b98ed83036875 WatchSource:0}: Error finding container 634cb55605d82d7b822507335571ed3815d125e7ba432588d02b98ed83036875: Status 404 returned error can't find the container with id 634cb55605d82d7b822507335571ed3815d125e7ba432588d02b98ed83036875
Oct 30 13:00:01.428809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.428769    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"]
Oct 30 13:00:01.429609 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:00:01.429583    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a68d1f1_747c_4b69_a51b_6f29cfc93212.slice/crio-b8f3a5f53076360a0add30927e32a6a433170ee815abd2322ff409e5fab85029 WatchSource:0}: Error finding container b8f3a5f53076360a0add30927e32a6a433170ee815abd2322ff409e5fab85029: Status 404 returned error can't find the container with id b8f3a5f53076360a0add30927e32a6a433170ee815abd2322ff409e5fab85029
Oct 30 13:00:01.679585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.679551    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" event={"ID":"6f8d98a9-d3fd-43d9-8334-f630bd89606c","Type":"ContainerStarted","Data":"41231c9b67f2c6ada7b67c144605b67bf5badbb4328c244fd22535f8387245e2"}
Oct 30 13:00:01.681425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.681400    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" event={"ID":"0fed7f4c-7755-49ef-ab10-0ee91cfd903d","Type":"ContainerStarted","Data":"7916359e1d3b1d0399b99ce59325407847f46487bbdc60056fcff3254c5bd983"}
Oct 30 13:00:01.681543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.681448    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" event={"ID":"0fed7f4c-7755-49ef-ab10-0ee91cfd903d","Type":"ContainerStarted","Data":"0e0002a674f10cc1d6815fbee837a0c428c72d50418abac0455de5525eefb2cd"}
Oct 30 13:00:01.682816 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.682793    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2" event={"ID":"94059e04-7219-4019-bd04-2c98bc58d2fb","Type":"ContainerStarted","Data":"634cb55605d82d7b822507335571ed3815d125e7ba432588d02b98ed83036875"}
Oct 30 13:00:01.693335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.693308    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" event={"ID":"439c6141-1827-4c42-b736-49e291f4672c","Type":"ContainerStarted","Data":"cb8233717e74c499019b58990018d005e555ee22ba2c189cdee3ae7fe8874393"}
Oct 30 13:00:01.693450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.693357    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" event={"ID":"8a68d1f1-747c-4b69-a51b-6f29cfc93212","Type":"ContainerStarted","Data":"b8f3a5f53076360a0add30927e32a6a433170ee815abd2322ff409e5fab85029"}
Oct 30 13:00:01.719953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.719907    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" podStartSLOduration=1.719894949 podStartE2EDuration="1.719894949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:01.718328528 +0000 UTC m=+1666.194048137" watchObservedRunningTime="2026-10-30 13:00:01.719894949 +0000 UTC m=+1666.195614559"
Oct 30 13:00:01.762863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.762812    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" podStartSLOduration=1.762794121 podStartE2EDuration="1.762794121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:01.758471537 +0000 UTC m=+1666.234191149" watchObservedRunningTime="2026-10-30 13:00:01.762794121 +0000 UTC m=+1666.238513731"
Oct 30 13:00:01.802931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:01.802872    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" podStartSLOduration=1.802857875 podStartE2EDuration="1.802857875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:01.797691407 +0000 UTC m=+1666.273411016" watchObservedRunningTime="2026-10-30 13:00:01.802857875 +0000 UTC m=+1666.278577476"
Oct 30 13:00:02.697101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.697000    4993 generic.go:356] "Generic (PLEG): container finished" podID="439c6141-1827-4c42-b736-49e291f4672c" containerID="7b3fa6b979f4dd8099d303f2dd1aa83fc5989ec5e44c5c8e59932aebccf84f44" exitCode=60
Oct 30 13:00:02.697609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.697110    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" event={"ID":"439c6141-1827-4c42-b736-49e291f4672c","Type":"ContainerDied","Data":"7b3fa6b979f4dd8099d303f2dd1aa83fc5989ec5e44c5c8e59932aebccf84f44"}
Oct 30 13:00:02.697609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.697431    4993 scope.go:122] "RemoveContainer" containerID="7b3fa6b979f4dd8099d303f2dd1aa83fc5989ec5e44c5c8e59932aebccf84f44"
Oct 30 13:00:02.699397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.699307    4993 generic.go:356] "Generic (PLEG): container finished" podID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerID="eb0e98bc7a84281b0aa8868ddcdeba7488e036de7ce1522699d241b0545f8285" exitCode=60
Oct 30 13:00:02.699397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.699336    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" event={"ID":"8a68d1f1-747c-4b69-a51b-6f29cfc93212","Type":"ContainerDied","Data":"eb0e98bc7a84281b0aa8868ddcdeba7488e036de7ce1522699d241b0545f8285"}
Oct 30 13:00:02.699742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.699716    4993 scope.go:122] "RemoveContainer" containerID="eb0e98bc7a84281b0aa8868ddcdeba7488e036de7ce1522699d241b0545f8285"
Oct 30 13:00:02.707164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.707135    4993 generic.go:356] "Generic (PLEG): container finished" podID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerID="a42bd0c8d5c00ed230ceb1deab04fddf14f0307b381a172aa610efae1880fc10" exitCode=60
Oct 30 13:00:02.707352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.707237    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" event={"ID":"6f8d98a9-d3fd-43d9-8334-f630bd89606c","Type":"ContainerDied","Data":"a42bd0c8d5c00ed230ceb1deab04fddf14f0307b381a172aa610efae1880fc10"}
Oct 30 13:00:02.707612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.707591    4993 scope.go:122] "RemoveContainer" containerID="a42bd0c8d5c00ed230ceb1deab04fddf14f0307b381a172aa610efae1880fc10"
Oct 30 13:00:02.709420 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.709394    4993 generic.go:356] "Generic (PLEG): container finished" podID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerID="7916359e1d3b1d0399b99ce59325407847f46487bbdc60056fcff3254c5bd983" exitCode=7
Oct 30 13:00:02.709573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.709555    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" event={"ID":"0fed7f4c-7755-49ef-ab10-0ee91cfd903d","Type":"ContainerDied","Data":"7916359e1d3b1d0399b99ce59325407847f46487bbdc60056fcff3254c5bd983"}
Oct 30 13:00:02.709787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.709766    4993 scope.go:122] "RemoveContainer" containerID="7916359e1d3b1d0399b99ce59325407847f46487bbdc60056fcff3254c5bd983"
Oct 30 13:00:02.712063 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.712041    4993 generic.go:356] "Generic (PLEG): container finished" podID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerID="39c4104d22cfc35732245e5ac6a4087acf7a76acba920cae622e8b1ed2664ad9" exitCode=60
Oct 30 13:00:02.712163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.712147    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2" event={"ID":"94059e04-7219-4019-bd04-2c98bc58d2fb","Type":"ContainerDied","Data":"39c4104d22cfc35732245e5ac6a4087acf7a76acba920cae622e8b1ed2664ad9"}
Oct 30 13:00:02.712500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:02.712483    4993 scope.go:122] "RemoveContainer" containerID="39c4104d22cfc35732245e5ac6a4087acf7a76acba920cae622e8b1ed2664ad9"
Oct 30 13:00:03.722319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.722277    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" event={"ID":"6f8d98a9-d3fd-43d9-8334-f630bd89606c","Type":"ContainerStarted","Data":"fde9b49283d0af0d7637f6810b45c5088d994255799e4c8a494ec79cdd8d8909"}
Oct 30 13:00:03.724679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.724650    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" event={"ID":"0fed7f4c-7755-49ef-ab10-0ee91cfd903d","Type":"ContainerStarted","Data":"6dbc0f6e1689e281974e349ba4e33faa07db08ee928167b0f67e3e022bcc59fb"}
Oct 30 13:00:03.727320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.727293    4993 generic.go:356] "Generic (PLEG): container finished" podID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerID="37ef2e681d833b84308881a82288c25caca618660259b98e5302980ef62f882b" exitCode=60
Oct 30 13:00:03.727834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.727432    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2" event={"ID":"94059e04-7219-4019-bd04-2c98bc58d2fb","Type":"ContainerDied","Data":"37ef2e681d833b84308881a82288c25caca618660259b98e5302980ef62f882b"}
Oct 30 13:00:03.727985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.727815    4993 scope.go:122] "RemoveContainer" containerID="37ef2e681d833b84308881a82288c25caca618660259b98e5302980ef62f882b"
Oct 30 13:00:03.728048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.728012    4993 scope.go:122] "RemoveContainer" containerID="39c4104d22cfc35732245e5ac6a4087acf7a76acba920cae622e8b1ed2664ad9"
Oct 30 13:00:03.728271 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:03.728245    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2_odh-ai-gateway-infra(94059e04-7219-4019-bd04-2c98bc58d2fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb"
Oct 30 13:00:03.730384 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.730361    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" event={"ID":"439c6141-1827-4c42-b736-49e291f4672c","Type":"ContainerStarted","Data":"404438dec98e1ae07235a2757aa0c2eb9646d819d59061e63a3873bf9db5bef8"}
Oct 30 13:00:03.732628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.732603    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" event={"ID":"8a68d1f1-747c-4b69-a51b-6f29cfc93212","Type":"ContainerStarted","Data":"116de7e76fbbfca09325e1c50dbf58e9c61f4ac74a065fcf1456df90caa89a46"}
Oct 30 13:00:03.810984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:03.810925    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" podStartSLOduration=3.810909579 podStartE2EDuration="3.810909579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:03.804947152 +0000 UTC m=+1668.280666763" watchObservedRunningTime="2026-10-30 13:00:03.810909579 +0000 UTC m=+1668.286629243"
Oct 30 13:00:04.756141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.756101    4993 generic.go:356] "Generic (PLEG): container finished" podID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerID="fde9b49283d0af0d7637f6810b45c5088d994255799e4c8a494ec79cdd8d8909" exitCode=60
Oct 30 13:00:04.756664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.756588    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" event={"ID":"6f8d98a9-d3fd-43d9-8334-f630bd89606c","Type":"ContainerDied","Data":"fde9b49283d0af0d7637f6810b45c5088d994255799e4c8a494ec79cdd8d8909"}
Oct 30 13:00:04.756664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.756597    4993 scope.go:122] "RemoveContainer" containerID="fde9b49283d0af0d7637f6810b45c5088d994255799e4c8a494ec79cdd8d8909"
Oct 30 13:00:04.756833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.756814    4993 scope.go:122] "RemoveContainer" containerID="a42bd0c8d5c00ed230ceb1deab04fddf14f0307b381a172aa610efae1880fc10"
Oct 30 13:00:04.756917 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:04.756894    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l_odh-ai-gateway-infra(6f8d98a9-d3fd-43d9-8334-f630bd89606c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c"
Oct 30 13:00:04.759452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.759407    4993 generic.go:356] "Generic (PLEG): container finished" podID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerID="6dbc0f6e1689e281974e349ba4e33faa07db08ee928167b0f67e3e022bcc59fb" exitCode=7
Oct 30 13:00:04.759566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.759544    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" event={"ID":"0fed7f4c-7755-49ef-ab10-0ee91cfd903d","Type":"ContainerDied","Data":"6dbc0f6e1689e281974e349ba4e33faa07db08ee928167b0f67e3e022bcc59fb"}
Oct 30 13:00:04.759778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.759762    4993 scope.go:122] "RemoveContainer" containerID="6dbc0f6e1689e281974e349ba4e33faa07db08ee928167b0f67e3e022bcc59fb"
Oct 30 13:00:04.760023 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:04.759998    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf_odh-ai-gateway-infra(0fed7f4c-7755-49ef-ab10-0ee91cfd903d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d"
Oct 30 13:00:04.761799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.761779    4993 scope.go:122] "RemoveContainer" containerID="37ef2e681d833b84308881a82288c25caca618660259b98e5302980ef62f882b"
Oct 30 13:00:04.762043 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:04.762022    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2_odh-ai-gateway-infra(94059e04-7219-4019-bd04-2c98bc58d2fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb"
Oct 30 13:00:04.763510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.763487    4993 generic.go:356] "Generic (PLEG): container finished" podID="439c6141-1827-4c42-b736-49e291f4672c" containerID="404438dec98e1ae07235a2757aa0c2eb9646d819d59061e63a3873bf9db5bef8" exitCode=60
Oct 30 13:00:04.763591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.763572    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" event={"ID":"439c6141-1827-4c42-b736-49e291f4672c","Type":"ContainerDied","Data":"404438dec98e1ae07235a2757aa0c2eb9646d819d59061e63a3873bf9db5bef8"}
Oct 30 13:00:04.763808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.763788    4993 scope.go:122] "RemoveContainer" containerID="404438dec98e1ae07235a2757aa0c2eb9646d819d59061e63a3873bf9db5bef8"
Oct 30 13:00:04.764059 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:04.764039    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn_odh-ai-gateway-infra(439c6141-1827-4c42-b736-49e291f4672c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" podUID="439c6141-1827-4c42-b736-49e291f4672c"
Oct 30 13:00:04.766170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.766148    4993 generic.go:356] "Generic (PLEG): container finished" podID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerID="116de7e76fbbfca09325e1c50dbf58e9c61f4ac74a065fcf1456df90caa89a46" exitCode=60
Oct 30 13:00:04.766284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.766235    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" event={"ID":"8a68d1f1-747c-4b69-a51b-6f29cfc93212","Type":"ContainerDied","Data":"116de7e76fbbfca09325e1c50dbf58e9c61f4ac74a065fcf1456df90caa89a46"}
Oct 30 13:00:04.766507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.766489    4993 scope.go:122] "RemoveContainer" containerID="116de7e76fbbfca09325e1c50dbf58e9c61f4ac74a065fcf1456df90caa89a46"
Oct 30 13:00:04.766691 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:04.766674    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9_odh-ai-gateway-infra(8a68d1f1-747c-4b69-a51b-6f29cfc93212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212"
Oct 30 13:00:04.772352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.772331    4993 scope.go:122] "RemoveContainer" containerID="7916359e1d3b1d0399b99ce59325407847f46487bbdc60056fcff3254c5bd983"
Oct 30 13:00:04.788427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.788398    4993 scope.go:122] "RemoveContainer" containerID="7b3fa6b979f4dd8099d303f2dd1aa83fc5989ec5e44c5c8e59932aebccf84f44"
Oct 30 13:00:04.808684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:04.808464    4993 scope.go:122] "RemoveContainer" containerID="eb0e98bc7a84281b0aa8868ddcdeba7488e036de7ce1522699d241b0545f8285"
Oct 30 13:00:05.780349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:05.780319    4993 scope.go:122] "RemoveContainer" containerID="404438dec98e1ae07235a2757aa0c2eb9646d819d59061e63a3873bf9db5bef8"
Oct 30 13:00:05.780754 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:05.780671    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn_odh-ai-gateway-infra(439c6141-1827-4c42-b736-49e291f4672c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" podUID="439c6141-1827-4c42-b736-49e291f4672c"
Oct 30 13:00:05.782106 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:05.782083    4993 scope.go:122] "RemoveContainer" containerID="116de7e76fbbfca09325e1c50dbf58e9c61f4ac74a065fcf1456df90caa89a46"
Oct 30 13:00:05.782452 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:05.782425    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9_odh-ai-gateway-infra(8a68d1f1-747c-4b69-a51b-6f29cfc93212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212"
Oct 30 13:00:05.783767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:05.783751    4993 scope.go:122] "RemoveContainer" containerID="fde9b49283d0af0d7637f6810b45c5088d994255799e4c8a494ec79cdd8d8909"
Oct 30 13:00:05.784043 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:05.784010    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l_odh-ai-gateway-infra(6f8d98a9-d3fd-43d9-8334-f630bd89606c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c"
Oct 30 13:00:05.785426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:05.785408    4993 scope.go:122] "RemoveContainer" containerID="6dbc0f6e1689e281974e349ba4e33faa07db08ee928167b0f67e3e022bcc59fb"
Oct 30 13:00:05.785680 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:05.785658    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf_odh-ai-gateway-infra(0fed7f4c-7755-49ef-ab10-0ee91cfd903d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d"
Oct 30 13:00:08.570582 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:08.570539    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:00:15.792264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:15.767166    4993 scope.go:122] "RemoveContainer" containerID="404438dec98e1ae07235a2757aa0c2eb9646d819d59061e63a3873bf9db5bef8"
Oct 30 13:00:15.990232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:15.990046    4993 scope.go:122] "RemoveContainer" containerID="cc0de15f2572d7529c527b9f479462c26f881f0d73466ff370aa9e2e5ed98ce1"
Oct 30 13:00:16.254228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:16.253488    4993 scope.go:122] "RemoveContainer" containerID="9bf8dcaf76073520ecf488d7f30c76a86bb27fb973234b9453557e89d420fa1d"
Oct 30 13:00:16.359822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:16.359790    4993 scope.go:122] "RemoveContainer" containerID="7a97d2ce5b073f16b9416d933935be4941a11bc553fc86faae385c492f2003f0"
Oct 30 13:00:16.457743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:16.449390    4993 scope.go:122] "RemoveContainer" containerID="d53b74b58f635bd682412a766bdca444391c920516c7147d9c5156b54f168973"
Oct 30 13:00:16.695229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:16.693839    4993 scope.go:122] "RemoveContainer" containerID="116de7e76fbbfca09325e1c50dbf58e9c61f4ac74a065fcf1456df90caa89a46"
Oct 30 13:00:16.695229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:16.694686    4993 scope.go:122] "RemoveContainer" containerID="6dbc0f6e1689e281974e349ba4e33faa07db08ee928167b0f67e3e022bcc59fb"
Oct 30 13:00:17.097231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:17.088172    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" event={"ID":"439c6141-1827-4c42-b736-49e291f4672c","Type":"ContainerStarted","Data":"63db4124050184adf6fd5f016009e18c734a1b49ffe23b523bb2aea5f8748a89"}
Oct 30 13:00:17.684225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:17.684173    4993 scope.go:122] "RemoveContainer" containerID="fde9b49283d0af0d7637f6810b45c5088d994255799e4c8a494ec79cdd8d8909"
Oct 30 13:00:17.684421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:17.684288    4993 scope.go:122] "RemoveContainer" containerID="37ef2e681d833b84308881a82288c25caca618660259b98e5302980ef62f882b"
Oct 30 13:00:18.099613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:18.099577    4993 generic.go:356] "Generic (PLEG): container finished" podID="439c6141-1827-4c42-b736-49e291f4672c" containerID="63db4124050184adf6fd5f016009e18c734a1b49ffe23b523bb2aea5f8748a89" exitCode=60
Oct 30 13:00:18.100069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:18.099661    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" event={"ID":"439c6141-1827-4c42-b736-49e291f4672c","Type":"ContainerDied","Data":"63db4124050184adf6fd5f016009e18c734a1b49ffe23b523bb2aea5f8748a89"}
Oct 30 13:00:18.100069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:18.099740    4993 scope.go:122] "RemoveContainer" containerID="404438dec98e1ae07235a2757aa0c2eb9646d819d59061e63a3873bf9db5bef8"
Oct 30 13:00:18.100201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:18.100179    4993 scope.go:122] "RemoveContainer" containerID="63db4124050184adf6fd5f016009e18c734a1b49ffe23b523bb2aea5f8748a89"
Oct 30 13:00:18.100621 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:18.100598    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn_odh-ai-gateway-infra(439c6141-1827-4c42-b736-49e291f4672c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" podUID="439c6141-1827-4c42-b736-49e291f4672c"
Oct 30 13:00:18.102086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:18.102064    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" event={"ID":"8a68d1f1-747c-4b69-a51b-6f29cfc93212","Type":"ContainerStarted","Data":"e563f947990d9235b0a617bfaa1141cc094b00a1686387e06416d5b2d182ab08"}
Oct 30 13:00:18.104399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:18.104378    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" event={"ID":"0fed7f4c-7755-49ef-ab10-0ee91cfd903d","Type":"ContainerStarted","Data":"36d731c04f0815a14fc8ff4703bdc12dd62e87bf3c3ebcc11f28c3bd4bb342b9"}
Oct 30 13:00:18.170830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:18.170803    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"]
Oct 30 13:00:19.114857 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.114823    4993 generic.go:356] "Generic (PLEG): container finished" podID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerID="3795f97fb09f81680484f0777ad8faa0f5a5ee04319f22e0eb6d4da51cd4a27c" exitCode=60
Oct 30 13:00:19.115299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.114903    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2" event={"ID":"94059e04-7219-4019-bd04-2c98bc58d2fb","Type":"ContainerDied","Data":"3795f97fb09f81680484f0777ad8faa0f5a5ee04319f22e0eb6d4da51cd4a27c"}
Oct 30 13:00:19.115299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.114972    4993 scope.go:122] "RemoveContainer" containerID="37ef2e681d833b84308881a82288c25caca618660259b98e5302980ef62f882b"
Oct 30 13:00:19.115494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.115472    4993 scope.go:122] "RemoveContainer" containerID="3795f97fb09f81680484f0777ad8faa0f5a5ee04319f22e0eb6d4da51cd4a27c"
Oct 30 13:00:19.115862 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:19.115837    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2_odh-ai-gateway-infra(94059e04-7219-4019-bd04-2c98bc58d2fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb"
Oct 30 13:00:19.119572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.119551    4993 generic.go:356] "Generic (PLEG): container finished" podID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerID="e563f947990d9235b0a617bfaa1141cc094b00a1686387e06416d5b2d182ab08" exitCode=60
Oct 30 13:00:19.119658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.119622    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" event={"ID":"8a68d1f1-747c-4b69-a51b-6f29cfc93212","Type":"ContainerDied","Data":"e563f947990d9235b0a617bfaa1141cc094b00a1686387e06416d5b2d182ab08"}
Oct 30 13:00:19.120034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.120015    4993 scope.go:122] "RemoveContainer" containerID="e563f947990d9235b0a617bfaa1141cc094b00a1686387e06416d5b2d182ab08"
Oct 30 13:00:19.120430 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:19.120403    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9_odh-ai-gateway-infra(8a68d1f1-747c-4b69-a51b-6f29cfc93212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212"
Oct 30 13:00:19.122168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.122147    4993 generic.go:356] "Generic (PLEG): container finished" podID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerID="838d38293060b69d18211d95371478882349833d8f9f7b1f5723d8915fdfd84f" exitCode=60
Oct 30 13:00:19.122298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.122237    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" event={"ID":"6f8d98a9-d3fd-43d9-8334-f630bd89606c","Type":"ContainerDied","Data":"838d38293060b69d18211d95371478882349833d8f9f7b1f5723d8915fdfd84f"}
Oct 30 13:00:19.122627 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.122604    4993 scope.go:122] "RemoveContainer" containerID="838d38293060b69d18211d95371478882349833d8f9f7b1f5723d8915fdfd84f"
Oct 30 13:00:19.122993 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:19.122967    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l_odh-ai-gateway-infra(6f8d98a9-d3fd-43d9-8334-f630bd89606c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c"
Oct 30 13:00:19.130351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.130323    4993 generic.go:356] "Generic (PLEG): container finished" podID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerID="36d731c04f0815a14fc8ff4703bdc12dd62e87bf3c3ebcc11f28c3bd4bb342b9" exitCode=60
Oct 30 13:00:19.130485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.130399    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" event={"ID":"0fed7f4c-7755-49ef-ab10-0ee91cfd903d","Type":"ContainerDied","Data":"36d731c04f0815a14fc8ff4703bdc12dd62e87bf3c3ebcc11f28c3bd4bb342b9"}
Oct 30 13:00:19.130812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.130790    4993 scope.go:122] "RemoveContainer" containerID="36d731c04f0815a14fc8ff4703bdc12dd62e87bf3c3ebcc11f28c3bd4bb342b9"
Oct 30 13:00:19.131160 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:19.131135    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf_odh-ai-gateway-infra(0fed7f4c-7755-49ef-ab10-0ee91cfd903d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d"
Oct 30 13:00:19.137262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.137242    4993 scope.go:122] "RemoveContainer" containerID="116de7e76fbbfca09325e1c50dbf58e9c61f4ac74a065fcf1456df90caa89a46"
Oct 30 13:00:19.158288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.157400    4993 scope.go:122] "RemoveContainer" containerID="fde9b49283d0af0d7637f6810b45c5088d994255799e4c8a494ec79cdd8d8909"
Oct 30 13:00:19.195566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.193440    4993 scope.go:122] "RemoveContainer" containerID="6dbc0f6e1689e281974e349ba4e33faa07db08ee928167b0f67e3e022bcc59fb"
Oct 30 13:00:19.240997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.240899    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"]
Oct 30 13:00:19.367554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.367476    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"]
Oct 30 13:00:19.396252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.394433    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:19.481697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.481609    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439c6141-1827-4c42-b736-49e291f4672c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439c6141-1827-4c42-b736-49e291f4672c-maas-api-service-ca\") pod \"439c6141-1827-4c42-b736-49e291f4672c\" (UID: \"439c6141-1827-4c42-b736-49e291f4672c\") "
Oct 30 13:00:19.481881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.481758    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439c6141-1827-4c42-b736-49e291f4672c-kube-api-access-5plch\" (UniqueName: \"kubernetes.io/projected/439c6141-1827-4c42-b736-49e291f4672c-kube-api-access-5plch\") pod \"439c6141-1827-4c42-b736-49e291f4672c\" (UID: \"439c6141-1827-4c42-b736-49e291f4672c\") "
Oct 30 13:00:19.482155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.482113    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439c6141-1827-4c42-b736-49e291f4672c-maas-api-service-ca" pod "439c6141-1827-4c42-b736-49e291f4672c" (UID: "439c6141-1827-4c42-b736-49e291f4672c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:19.482767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.482740    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/439c6141-1827-4c42-b736-49e291f4672c-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:19.486748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.484786    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439c6141-1827-4c42-b736-49e291f4672c-kube-api-access-5plch" pod "439c6141-1827-4c42-b736-49e291f4672c" (UID: "439c6141-1827-4c42-b736-49e291f4672c"). InnerVolumeSpecName "kube-api-access-5plch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:19.587230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:19.586373    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5plch\" (UniqueName: \"kubernetes.io/projected/439c6141-1827-4c42-b736-49e291f4672c-kube-api-access-5plch\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:20.141675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.141624    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn" event={"ID":"439c6141-1827-4c42-b736-49e291f4672c","Type":"ContainerDied","Data":"cb8233717e74c499019b58990018d005e555ee22ba2c189cdee3ae7fe8874393"}
Oct 30 13:00:20.142170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.141703    4993 scope.go:122] "RemoveContainer" containerID="63db4124050184adf6fd5f016009e18c734a1b49ffe23b523bb2aea5f8748a89"
Oct 30 13:00:20.142170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.142032    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"
Oct 30 13:00:20.210335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.210298    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"]
Oct 30 13:00:20.312173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.312141    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"]
Oct 30 13:00:20.447954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.443255    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"]
Oct 30 13:00:20.447954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.443325    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-87cae1-29889420-c7fmn"]
Oct 30 13:00:20.717331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.717286    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:20.807264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.807228    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-kube-api-access-zqcb7\" (UniqueName: \"kubernetes.io/projected/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-kube-api-access-zqcb7\") pod \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\" (UID: \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\") "
Oct 30 13:00:20.809184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.809156    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-maas-api-service-ca\") pod \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\" (UID: \"0fed7f4c-7755-49ef-ab10-0ee91cfd903d\") "
Oct 30 13:00:20.818335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.811946    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-kube-api-access-zqcb7" pod "0fed7f4c-7755-49ef-ab10-0ee91cfd903d" (UID: "0fed7f4c-7755-49ef-ab10-0ee91cfd903d"). InnerVolumeSpecName "kube-api-access-zqcb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:20.820469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.820420    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-maas-api-service-ca" pod "0fed7f4c-7755-49ef-ab10-0ee91cfd903d" (UID: "0fed7f4c-7755-49ef-ab10-0ee91cfd903d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:20.826378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.826084    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:20.828946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.828922    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:20.834444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.834417    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:20.920758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.920722    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a68d1f1-747c-4b69-a51b-6f29cfc93212-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68d1f1-747c-4b69-a51b-6f29cfc93212-maas-api-service-ca\") pod \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\" (UID: \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\") "
Oct 30 13:00:20.921531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.921506    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94059e04-7219-4019-bd04-2c98bc58d2fb-kube-api-access-b68qd\" (UniqueName: \"kubernetes.io/projected/94059e04-7219-4019-bd04-2c98bc58d2fb-kube-api-access-b68qd\") pod \"94059e04-7219-4019-bd04-2c98bc58d2fb\" (UID: \"94059e04-7219-4019-bd04-2c98bc58d2fb\") "
Oct 30 13:00:20.921758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.921739    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a68d1f1-747c-4b69-a51b-6f29cfc93212-kube-api-access-t94bb\" (UniqueName: \"kubernetes.io/projected/8a68d1f1-747c-4b69-a51b-6f29cfc93212-kube-api-access-t94bb\") pod \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\" (UID: \"8a68d1f1-747c-4b69-a51b-6f29cfc93212\") "
Oct 30 13:00:20.921902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.921885    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8d98a9-d3fd-43d9-8334-f630bd89606c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d98a9-d3fd-43d9-8334-f630bd89606c-maas-api-service-ca\") pod \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\" (UID: \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\") "
Oct 30 13:00:20.922161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.922140    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94059e04-7219-4019-bd04-2c98bc58d2fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94059e04-7219-4019-bd04-2c98bc58d2fb-maas-api-service-ca\") pod \"94059e04-7219-4019-bd04-2c98bc58d2fb\" (UID: \"94059e04-7219-4019-bd04-2c98bc58d2fb\") "
Oct 30 13:00:20.922368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.922351    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8d98a9-d3fd-43d9-8334-f630bd89606c-kube-api-access-2m8jg\" (UniqueName: \"kubernetes.io/projected/6f8d98a9-d3fd-43d9-8334-f630bd89606c-kube-api-access-2m8jg\") pod \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\" (UID: \"6f8d98a9-d3fd-43d9-8334-f630bd89606c\") "
Oct 30 13:00:20.923108 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.923084    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcb7\" (UniqueName: \"kubernetes.io/projected/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-kube-api-access-zqcb7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:20.923318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.923301    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fed7f4c-7755-49ef-ab10-0ee91cfd903d-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:20.925608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.921438    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a68d1f1-747c-4b69-a51b-6f29cfc93212-maas-api-service-ca" pod "8a68d1f1-747c-4b69-a51b-6f29cfc93212" (UID: "8a68d1f1-747c-4b69-a51b-6f29cfc93212"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:20.925729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.925542    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8d98a9-d3fd-43d9-8334-f630bd89606c-maas-api-service-ca" pod "6f8d98a9-d3fd-43d9-8334-f630bd89606c" (UID: "6f8d98a9-d3fd-43d9-8334-f630bd89606c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:20.925836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.925742    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94059e04-7219-4019-bd04-2c98bc58d2fb-maas-api-service-ca" pod "94059e04-7219-4019-bd04-2c98bc58d2fb" (UID: "94059e04-7219-4019-bd04-2c98bc58d2fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:20.927551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.927503    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a68d1f1-747c-4b69-a51b-6f29cfc93212-kube-api-access-t94bb" pod "8a68d1f1-747c-4b69-a51b-6f29cfc93212" (UID: "8a68d1f1-747c-4b69-a51b-6f29cfc93212"). InnerVolumeSpecName "kube-api-access-t94bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:20.928170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.928147    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94059e04-7219-4019-bd04-2c98bc58d2fb-kube-api-access-b68qd" pod "94059e04-7219-4019-bd04-2c98bc58d2fb" (UID: "94059e04-7219-4019-bd04-2c98bc58d2fb"). InnerVolumeSpecName "kube-api-access-b68qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:20.928575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:20.928547    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8d98a9-d3fd-43d9-8334-f630bd89606c-kube-api-access-2m8jg" pod "6f8d98a9-d3fd-43d9-8334-f630bd89606c" (UID: "6f8d98a9-d3fd-43d9-8334-f630bd89606c"). InnerVolumeSpecName "kube-api-access-2m8jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:21.025035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.024990    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d98a9-d3fd-43d9-8334-f630bd89606c-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.025035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.025029    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94059e04-7219-4019-bd04-2c98bc58d2fb-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.025035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.025044    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m8jg\" (UniqueName: \"kubernetes.io/projected/6f8d98a9-d3fd-43d9-8334-f630bd89606c-kube-api-access-2m8jg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.025416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.025057    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a68d1f1-747c-4b69-a51b-6f29cfc93212-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.025416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.025069    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b68qd\" (UniqueName: \"kubernetes.io/projected/94059e04-7219-4019-bd04-2c98bc58d2fb-kube-api-access-b68qd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.025416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.025080    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t94bb\" (UniqueName: \"kubernetes.io/projected/8a68d1f1-747c-4b69-a51b-6f29cfc93212-kube-api-access-t94bb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.167269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.167232    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"
Oct 30 13:00:21.167760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.167265    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2" event={"ID":"94059e04-7219-4019-bd04-2c98bc58d2fb","Type":"ContainerDied","Data":"634cb55605d82d7b822507335571ed3815d125e7ba432588d02b98ed83036875"}
Oct 30 13:00:21.167760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.167337    4993 scope.go:122] "RemoveContainer" containerID="3795f97fb09f81680484f0777ad8faa0f5a5ee04319f22e0eb6d4da51cd4a27c"
Oct 30 13:00:21.172241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.171271    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9" event={"ID":"8a68d1f1-747c-4b69-a51b-6f29cfc93212","Type":"ContainerDied","Data":"b8f3a5f53076360a0add30927e32a6a433170ee815abd2322ff409e5fab85029"}
Oct 30 13:00:21.172241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.171380    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"
Oct 30 13:00:21.187897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.186979    4993 scope.go:122] "RemoveContainer" containerID="e563f947990d9235b0a617bfaa1141cc094b00a1686387e06416d5b2d182ab08"
Oct 30 13:00:21.189285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.189262    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"
Oct 30 13:00:21.189412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.189373    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l" event={"ID":"6f8d98a9-d3fd-43d9-8334-f630bd89606c","Type":"ContainerDied","Data":"41231c9b67f2c6ada7b67c144605b67bf5badbb4328c244fd22535f8387245e2"}
Oct 30 13:00:21.191858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.191834    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf" event={"ID":"0fed7f4c-7755-49ef-ab10-0ee91cfd903d","Type":"ContainerDied","Data":"0e0002a674f10cc1d6815fbee837a0c428c72d50418abac0455de5525eefb2cd"}
Oct 30 13:00:21.191977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.191921    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"
Oct 30 13:00:21.203886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.202956    4993 scope.go:122] "RemoveContainer" containerID="838d38293060b69d18211d95371478882349833d8f9f7b1f5723d8915fdfd84f"
Oct 30 13:00:21.218155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.218090    4993 scope.go:122] "RemoveContainer" containerID="36d731c04f0815a14fc8ff4703bdc12dd62e87bf3c3ebcc11f28c3bd4bb342b9"
Oct 30 13:00:21.246827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.246791    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"]
Oct 30 13:00:21.251869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.251839    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889420-b6s7l"]
Oct 30 13:00:21.283290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.282650    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"]
Oct 30 13:00:21.287855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.287823    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889420-6jvx9"]
Oct 30 13:00:21.321414 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.321360    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"]
Oct 30 13:00:21.326300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.326267    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889420-lngm2"]
Oct 30 13:00:21.384419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.383060    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"]
Oct 30 13:00:21.405972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.405941    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ea117f29-29889420-4xxlf"]
Oct 30 13:00:21.693081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.693048    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" path="/var/lib/kubelet/pods/0fed7f4c-7755-49ef-ab10-0ee91cfd903d/volumes"
Oct 30 13:00:21.693617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.693594    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439c6141-1827-4c42-b736-49e291f4672c" path="/var/lib/kubelet/pods/439c6141-1827-4c42-b736-49e291f4672c/volumes"
Oct 30 13:00:21.693980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.693967    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" path="/var/lib/kubelet/pods/6f8d98a9-d3fd-43d9-8334-f630bd89606c/volumes"
Oct 30 13:00:21.694605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.694587    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" path="/var/lib/kubelet/pods/8a68d1f1-747c-4b69-a51b-6f29cfc93212/volumes"
Oct 30 13:00:21.695022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:21.695008    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" path="/var/lib/kubelet/pods/94059e04-7219-4019-bd04-2c98bc58d2fb/volumes"
Oct 30 13:00:33.042907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:33.042871    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"]
Oct 30 13:00:33.054254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:33.054223    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a32dec81-fmtjt"]
Oct 30 13:00:33.737229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:33.727239    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac3e73d-c73a-4fb8-8473-ba398564b702" path="/var/lib/kubelet/pods/0ac3e73d-c73a-4fb8-8473-ba398564b702/volumes"
Oct 30 13:00:38.572562 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:00:38.572520    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:00:40.035239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.035174    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"]
Oct 30 13:00:40.049901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.049869    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b09ffb25-z6gcg"]
Oct 30 13:00:40.797438 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.797392    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"]
Oct 30 13:00:40.798324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798294    4993 cpu_manager.go:414] "containerMap: removing container" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:40.798324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798322    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798351    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798361    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798373    4993 cpu_manager.go:414] "containerMap: removing container" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798381    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798399    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798406    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798418    4993 cpu_manager.go:414] "containerMap: removing container" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798425    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798445    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798452    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798470    4993 cpu_manager.go:414] "containerMap: removing container" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798477    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798494    4993 cpu_manager.go:414] "containerMap: removing container" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798501    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798512    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798519    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798539    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798546    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:40.798544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798556    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798564    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798729    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798740    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798750    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798767    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798777    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798786    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798796    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798805    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798817    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798825    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:40.799537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.798833    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:40.803577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.803550    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.812414 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.812386    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"]
Oct 30 13:00:40.824766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.824738    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4561d2-openshift-default-dockercfg-xvr9c\""
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963257    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-credential-socket\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963315    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-podinfo\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963364    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftcj\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-kube-api-access-kftcj\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963465    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-data\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963593    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istiod-ca-cert\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963731    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-token\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963772    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-envoy\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963826    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-socket\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:40.964243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:40.963882    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-certs\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.068951    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftcj\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-kube-api-access-kftcj\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.069069    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-data\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.069126    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istiod-ca-cert\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.069273    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-token\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.069323    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-envoy\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.069381    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-socket\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.069433    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-certs\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.069507    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-credential-socket\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.069534    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-podinfo\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.071177    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-data\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.072275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.072037    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istiod-ca-cert\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.083232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.078575    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-credential-socket\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.083232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.078613    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-certs\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.083232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.078752    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-socket\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.094649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.084098    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-envoy\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.120227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.105724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-token\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.120227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.107867    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-podinfo\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.125239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.124627    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftcj\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-kube-api-access-kftcj\") pod \"e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") " pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.125239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.124845    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:41.537115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.537083    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"]
Oct 30 13:00:41.541601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.541430    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:00:41.541601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.541520    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:00:41.541601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.541570    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:00:41.720106 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:41.720068    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf73034-530a-4278-8401-4b9700610c32" path="/var/lib/kubelet/pods/baf73034-530a-4278-8401-4b9700610c32/volumes"
Oct 30 13:00:42.527914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:42.527870    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" event={"ID":"1aa47a74-1ead-4407-8fab-2a84bf8f9172","Type":"ContainerStarted","Data":"05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2"}
Oct 30 13:00:42.528390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:42.527932    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" event={"ID":"1aa47a74-1ead-4407-8fab-2a84bf8f9172","Type":"ContainerStarted","Data":"3f3810ff558cd8a6e2a1caec50464520a9b59dd57f03ee6f3686f54632a2a2de"}
Oct 30 13:00:42.590237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:42.590165    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" podStartSLOduration=2.590149731 podStartE2EDuration="2.590149731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:42.587325049 +0000 UTC m=+1707.063044651" watchObservedRunningTime="2026-10-30 13:00:42.590149731 +0000 UTC m=+1707.065869341"
Oct 30 13:00:43.126065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:43.126027    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:43.132250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:43.132192    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:43.536203 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:43.536170    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:43.537269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:43.537245    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:00:46.952145 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.952104    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"]
Oct 30 13:00:46.953027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953002    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:46.953099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953028    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:46.953099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953049    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:46.953099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953057    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8d98a9-d3fd-43d9-8334-f630bd89606c" containerName="cleanup"
Oct 30 13:00:46.953099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953067    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:46.953099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953075    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:46.953362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953109    4993 cpu_manager.go:414] "containerMap: removing container" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:46.953362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953116    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:46.953362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953273    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="439c6141-1827-4c42-b736-49e291f4672c" containerName="cleanup"
Oct 30 13:00:46.953362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953291    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a68d1f1-747c-4b69-a51b-6f29cfc93212" containerName="cleanup"
Oct 30 13:00:46.953362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953301    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="94059e04-7219-4019-bd04-2c98bc58d2fb" containerName="cleanup"
Oct 30 13:00:46.953362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.953317    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fed7f4c-7755-49ef-ab10-0ee91cfd903d" containerName="cleanup"
Oct 30 13:00:46.958454 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.958430    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:46.980069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.980035    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"]
Oct 30 13:00:46.987604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.987563    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:00:46.987952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.987930    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:00:46.988190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.988174    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:00:46.988396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:46.988382    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:00:47.099169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.083955    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.099169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.098837    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.099169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.099008    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76zv\" (UniqueName: \"kubernetes.io/projected/ddd42e3d-c494-4217-b211-09954d68f58e-kube-api-access-c76zv\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.204161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.204059    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76zv\" (UniqueName: \"kubernetes.io/projected/ddd42e3d-c494-4217-b211-09954d68f58e-kube-api-access-c76zv\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.204392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.204223    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.204392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.204343    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.205874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.205298    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.227236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.225643    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.232183 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.232156    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76zv\" (UniqueName: \"kubernetes.io/projected/ddd42e3d-c494-4217-b211-09954d68f58e-kube-api-access-c76zv\") pod \"maas-api-revoke-keys-e2e-api-a-87cae1-krvj7\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.281667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.281635    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:47.592081 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:00:47.592042    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd42e3d_c494_4217_b211_09954d68f58e.slice/crio-81d4d27638d453c1053ba8c335a2a881ad9da75568214abb8f05ce6dc04003d5 WatchSource:0}: Error finding container 81d4d27638d453c1053ba8c335a2a881ad9da75568214abb8f05ce6dc04003d5: Status 404 returned error can't find the container with id 81d4d27638d453c1053ba8c335a2a881ad9da75568214abb8f05ce6dc04003d5
Oct 30 13:00:47.592496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:47.592471    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"]
Oct 30 13:00:48.033029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:48.032995    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"]
Oct 30 13:00:48.037449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:48.037424    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13d9f9eb-r8jcl"]
Oct 30 13:00:48.585397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:48.585353    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7" event={"ID":"ddd42e3d-c494-4217-b211-09954d68f58e","Type":"ContainerStarted","Data":"d1375fc6723bd879bd2fb8cd9c30a59f259c4d5efa650cfe20b8ff46f06ed0a9"}
Oct 30 13:00:48.585618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:48.585420    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7" event={"ID":"ddd42e3d-c494-4217-b211-09954d68f58e","Type":"ContainerStarted","Data":"81d4d27638d453c1053ba8c335a2a881ad9da75568214abb8f05ce6dc04003d5"}
Oct 30 13:00:48.628688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:48.628631    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7" podStartSLOduration=2.6286132330000003 podStartE2EDuration="2.628613233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:48.628254671 +0000 UTC m=+1713.103974281" watchObservedRunningTime="2026-10-30 13:00:48.628613233 +0000 UTC m=+1713.104332844"
Oct 30 13:00:49.595690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:49.595648    4993 generic.go:356] "Generic (PLEG): container finished" podID="ddd42e3d-c494-4217-b211-09954d68f58e" containerID="d1375fc6723bd879bd2fb8cd9c30a59f259c4d5efa650cfe20b8ff46f06ed0a9" exitCode=0
Oct 30 13:00:49.596162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:49.595728    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7" event={"ID":"ddd42e3d-c494-4217-b211-09954d68f58e","Type":"ContainerDied","Data":"d1375fc6723bd879bd2fb8cd9c30a59f259c4d5efa650cfe20b8ff46f06ed0a9"}
Oct 30 13:00:49.692403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:49.692368    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9dc4580-c625-4505-950d-e7579d3d2af9" path="/var/lib/kubelet/pods/a9dc4580-c625-4505-950d-e7579d3d2af9/volumes"
Oct 30 13:00:50.799476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.799448    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:50.847455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.847424    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-xks-ca\") pod \"ddd42e3d-c494-4217-b211-09954d68f58e\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") "
Oct 30 13:00:50.847639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.847490    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd42e3d-c494-4217-b211-09954d68f58e-kube-api-access-c76zv\" (UniqueName: \"kubernetes.io/projected/ddd42e3d-c494-4217-b211-09954d68f58e-kube-api-access-c76zv\") pod \"ddd42e3d-c494-4217-b211-09954d68f58e\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") "
Oct 30 13:00:50.847639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.847530    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-openshift-ca\") pod \"ddd42e3d-c494-4217-b211-09954d68f58e\" (UID: \"ddd42e3d-c494-4217-b211-09954d68f58e\") "
Oct 30 13:00:50.847904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.847873    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-openshift-ca" pod "ddd42e3d-c494-4217-b211-09954d68f58e" (UID: "ddd42e3d-c494-4217-b211-09954d68f58e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:50.848086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.848067    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:50.849726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.849703    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd42e3d-c494-4217-b211-09954d68f58e-kube-api-access-c76zv" pod "ddd42e3d-c494-4217-b211-09954d68f58e" (UID: "ddd42e3d-c494-4217-b211-09954d68f58e"). InnerVolumeSpecName "kube-api-access-c76zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:50.849976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.849941    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-xks-ca" pod "ddd42e3d-c494-4217-b211-09954d68f58e" (UID: "ddd42e3d-c494-4217-b211-09954d68f58e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:50.948997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.948964    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd42e3d-c494-4217-b211-09954d68f58e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:50.948997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:50.948994    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76zv\" (UniqueName: \"kubernetes.io/projected/ddd42e3d-c494-4217-b211-09954d68f58e-kube-api-access-c76zv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:51.032633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:51.032597    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"]
Oct 30 13:00:51.037522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:51.037491    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16434684-2vvqc"]
Oct 30 13:00:51.614698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:51.614666    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"
Oct 30 13:00:51.614698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:51.614688    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7" event={"ID":"ddd42e3d-c494-4217-b211-09954d68f58e","Type":"ContainerDied","Data":"81d4d27638d453c1053ba8c335a2a881ad9da75568214abb8f05ce6dc04003d5"}
Oct 30 13:00:51.614967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:51.614747    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d4d27638d453c1053ba8c335a2a881ad9da75568214abb8f05ce6dc04003d5"
Oct 30 13:00:51.696677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:51.696639    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43afe350-83fe-4157-9ebc-1f36b9a9d008" path="/var/lib/kubelet/pods/43afe350-83fe-4157-9ebc-1f36b9a9d008/volumes"
Oct 30 13:00:52.032906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:52.032872    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"]
Oct 30 13:00:52.037423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:52.037398    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ee3e44-7jk5c"]
Oct 30 13:00:53.695584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:53.695547    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6a311e-4140-468f-a44f-311e27151550" path="/var/lib/kubelet/pods/5b6a311e-4140-468f-a44f-311e27151550/volumes"
Oct 30 13:00:55.195507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.195472    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"]
Oct 30 13:00:55.196074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.196059    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ddd42e3d-c494-4217-b211-09954d68f58e" containerName="revoke-keys"
Oct 30 13:00:55.196074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.196074    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd42e3d-c494-4217-b211-09954d68f58e" containerName="revoke-keys"
Oct 30 13:00:55.196201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.196169    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd42e3d-c494-4217-b211-09954d68f58e" containerName="revoke-keys"
Oct 30 13:00:55.201524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.201503    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.219470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.219445    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"]
Oct 30 13:00:55.225366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.225337    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:00:55.225366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.225340    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:00:55.225571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.225420    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:00:55.230562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.230535    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:00:55.320024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.319994    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.320252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.320199    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7b4t\" (UniqueName: \"kubernetes.io/projected/2565a5c8-ca9c-4600-8cab-8b84aba412c5-kube-api-access-g7b4t\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.320366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.320334    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.421971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.421932    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.422139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.421998    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.422184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.422169    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7b4t\" (UniqueName: \"kubernetes.io/projected/2565a5c8-ca9c-4600-8cab-8b84aba412c5-kube-api-access-g7b4t\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.422791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.422771    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.424495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.424475    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.448553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.448488    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7b4t\" (UniqueName: \"kubernetes.io/projected/2565a5c8-ca9c-4600-8cab-8b84aba412c5-kube-api-access-g7b4t\") pod \"maas-api-revoke-sub-80970991-8js42\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.518664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.518630    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:55.823873 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:00:55.823839    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2565a5c8_ca9c_4600_8cab_8b84aba412c5.slice/crio-3dc052fbcce2b3de4686264d09bac8c6703925039bab0cf493a03dfcea5ffbb7 WatchSource:0}: Error finding container 3dc052fbcce2b3de4686264d09bac8c6703925039bab0cf493a03dfcea5ffbb7: Status 404 returned error can't find the container with id 3dc052fbcce2b3de4686264d09bac8c6703925039bab0cf493a03dfcea5ffbb7
Oct 30 13:00:55.824130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:55.824104    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"]
Oct 30 13:00:56.659959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:56.659921    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42" event={"ID":"2565a5c8-ca9c-4600-8cab-8b84aba412c5","Type":"ContainerStarted","Data":"aa855d2ede28fdfcd75266efd8df8ad3973a466bbdea7ff5dfb4a5891c96ea6c"}
Oct 30 13:00:56.660448 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:56.659976    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42" event={"ID":"2565a5c8-ca9c-4600-8cab-8b84aba412c5","Type":"ContainerStarted","Data":"3dc052fbcce2b3de4686264d09bac8c6703925039bab0cf493a03dfcea5ffbb7"}
Oct 30 13:00:56.704550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:56.704501    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42" podStartSLOduration=1.704486122 podStartE2EDuration="1.704486122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:56.699868229 +0000 UTC m=+1721.175587839" watchObservedRunningTime="2026-10-30 13:00:56.704486122 +0000 UTC m=+1721.180205729"
Oct 30 13:00:57.672916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:57.672876    4993 generic.go:356] "Generic (PLEG): container finished" podID="2565a5c8-ca9c-4600-8cab-8b84aba412c5" containerID="aa855d2ede28fdfcd75266efd8df8ad3973a466bbdea7ff5dfb4a5891c96ea6c" exitCode=0
Oct 30 13:00:57.673574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:57.673009    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42" event={"ID":"2565a5c8-ca9c-4600-8cab-8b84aba412c5","Type":"ContainerDied","Data":"aa855d2ede28fdfcd75266efd8df8ad3973a466bbdea7ff5dfb4a5891c96ea6c"}
Oct 30 13:00:58.841053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:58.841027    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:58.977592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:58.977563    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-openshift-ca\") pod \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") "
Oct 30 13:00:58.977783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:58.977667    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-xks-ca\") pod \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") "
Oct 30 13:00:58.977783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:58.977771    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2565a5c8-ca9c-4600-8cab-8b84aba412c5-kube-api-access-g7b4t\" (UniqueName: \"kubernetes.io/projected/2565a5c8-ca9c-4600-8cab-8b84aba412c5-kube-api-access-g7b4t\") pod \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\" (UID: \"2565a5c8-ca9c-4600-8cab-8b84aba412c5\") "
Oct 30 13:00:58.978018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:58.977990    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-openshift-ca" pod "2565a5c8-ca9c-4600-8cab-8b84aba412c5" (UID: "2565a5c8-ca9c-4600-8cab-8b84aba412c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:58.978301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:58.978279    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:58.979831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:58.979804    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-xks-ca" pod "2565a5c8-ca9c-4600-8cab-8b84aba412c5" (UID: "2565a5c8-ca9c-4600-8cab-8b84aba412c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:58.979936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:58.979853    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2565a5c8-ca9c-4600-8cab-8b84aba412c5-kube-api-access-g7b4t" pod "2565a5c8-ca9c-4600-8cab-8b84aba412c5" (UID: "2565a5c8-ca9c-4600-8cab-8b84aba412c5"). InnerVolumeSpecName "kube-api-access-g7b4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:59.080010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:59.079974    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2565a5c8-ca9c-4600-8cab-8b84aba412c5-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:59.080010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:59.080006    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7b4t\" (UniqueName: \"kubernetes.io/projected/2565a5c8-ca9c-4600-8cab-8b84aba412c5-kube-api-access-g7b4t\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:00:59.693369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:59.693335    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"
Oct 30 13:00:59.693590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:59.693558    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42" event={"ID":"2565a5c8-ca9c-4600-8cab-8b84aba412c5","Type":"ContainerDied","Data":"3dc052fbcce2b3de4686264d09bac8c6703925039bab0cf493a03dfcea5ffbb7"}
Oct 30 13:00:59.693674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:00:59.693617    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dc052fbcce2b3de4686264d09bac8c6703925039bab0cf493a03dfcea5ffbb7"
Oct 30 13:01:00.032997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:00.032960    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"]
Oct 30 13:01:00.045239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:00.045198    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6b51790-pfxwr"]
Oct 30 13:01:01.695321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:01.695288    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da156ee-bcee-44c3-821c-37b2870f8830" path="/var/lib/kubelet/pods/3da156ee-bcee-44c3-821c-37b2870f8830/volumes"
Oct 30 13:01:02.031201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:02.031154    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"]
Oct 30 13:01:02.045965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:02.045926    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa667de1-9dl24"]
Oct 30 13:01:03.694451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:03.694414    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e98edd3-7703-4368-8f6c-55d669f33787" path="/var/lib/kubelet/pods/4e98edd3-7703-4368-8f6c-55d669f33787/volumes"
Oct 30 13:01:08.571698 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:01:08.571653    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:01:21.192232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:21.191936    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:21.261234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:21.260316    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:21.302233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:21.301305    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:21.306231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:21.304777    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:01:21.617232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:21.616381    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:21.679231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:21.678404    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:21.728231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:21.727322    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:21.740241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:21.739418    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:22.091848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.059340    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"]
Oct 30 13:01:22.091848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.060254    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2565a5c8-ca9c-4600-8cab-8b84aba412c5" containerName="revoke-keys"
Oct 30 13:01:22.091848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.060276    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2565a5c8-ca9c-4600-8cab-8b84aba412c5" containerName="revoke-keys"
Oct 30 13:01:22.091848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.060481    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2565a5c8-ca9c-4600-8cab-8b84aba412c5" containerName="revoke-keys"
Oct 30 13:01:22.097571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.097497    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"]
Oct 30 13:01:22.097929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.097883    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.119635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.117300    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:22.127634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.126744    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:01:22.127634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.126814    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:01:22.127634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.126906    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:01:22.288233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.282690    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkw2g\" (UniqueName: \"kubernetes.io/projected/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-kube-api-access-kkw2g\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.288233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.282937    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.288233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.282982    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.386047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.385003    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.386047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.385056    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.386047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.385194    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkw2g\" (UniqueName: \"kubernetes.io/projected/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-kube-api-access-kkw2g\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.386047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.385896    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.388708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.388680    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.411349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.411291    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkw2g\" (UniqueName: \"kubernetes.io/projected/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-kube-api-access-kkw2g\") pod \"maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.441912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.440988    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:22.813141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:22.813057    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"]
Oct 30 13:01:22.816303 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:01:22.816262    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ce7766_a60f_4f37_b03d_cd36ed6a83f0.slice/crio-6fc1cfbb213711774005882589534615dc9d7faf79f88f1a9d2b724052912a09 WatchSource:0}: Error finding container 6fc1cfbb213711774005882589534615dc9d7faf79f88f1a9d2b724052912a09: Status 404 returned error can't find the container with id 6fc1cfbb213711774005882589534615dc9d7faf79f88f1a9d2b724052912a09
Oct 30 13:01:23.141751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:23.141702    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v" event={"ID":"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0","Type":"ContainerStarted","Data":"6fc1cfbb213711774005882589534615dc9d7faf79f88f1a9d2b724052912a09"}
Oct 30 13:01:24.174231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:24.174049    4993 generic.go:356] "Generic (PLEG): container finished" podID="d4ce7766-a60f-4f37-b03d-cd36ed6a83f0" containerID="470bfdfe8d2277ec9a622bb4090bd279b140f4ad7551c8d990c9186abf0de110" exitCode=0
Oct 30 13:01:24.174231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:24.174146    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v" event={"ID":"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0","Type":"ContainerDied","Data":"470bfdfe8d2277ec9a622bb4090bd279b140f4ad7551c8d990c9186abf0de110"}
Oct 30 13:01:25.598912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.598879    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:25.681706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.681666    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-openshift-ca\") pod \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") "
Oct 30 13:01:25.696544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.688607    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-openshift-ca" pod "d4ce7766-a60f-4f37-b03d-cd36ed6a83f0" (UID: "d4ce7766-a60f-4f37-b03d-cd36ed6a83f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:25.696813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.696787    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-kube-api-access-kkw2g\" (UniqueName: \"kubernetes.io/projected/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-kube-api-access-kkw2g\") pod \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") "
Oct 30 13:01:25.697694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.697662    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-xks-ca\") pod \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\" (UID: \"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0\") "
Oct 30 13:01:25.698589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.698562    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:25.709609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.709561    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-kube-api-access-kkw2g" pod "d4ce7766-a60f-4f37-b03d-cd36ed6a83f0" (UID: "d4ce7766-a60f-4f37-b03d-cd36ed6a83f0"). InnerVolumeSpecName "kube-api-access-kkw2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:25.712744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.712702    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-xks-ca" pod "d4ce7766-a60f-4f37-b03d-cd36ed6a83f0" (UID: "d4ce7766-a60f-4f37-b03d-cd36ed6a83f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:25.800681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.800580    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:25.800681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:25.800623    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkw2g\" (UniqueName: \"kubernetes.io/projected/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0-kube-api-access-kkw2g\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:26.215329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:26.214140    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v" event={"ID":"d4ce7766-a60f-4f37-b03d-cd36ed6a83f0","Type":"ContainerDied","Data":"6fc1cfbb213711774005882589534615dc9d7faf79f88f1a9d2b724052912a09"}
Oct 30 13:01:26.215329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:26.214225    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fc1cfbb213711774005882589534615dc9d7faf79f88f1a9d2b724052912a09"
Oct 30 13:01:26.215329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:26.214305    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"
Oct 30 13:01:26.588153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:26.588110    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"]
Oct 30 13:01:26.596218 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:26.596162    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87cae1-krvj7"]
Oct 30 13:01:27.698387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:27.698348    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd42e3d-c494-4217-b211-09954d68f58e" path="/var/lib/kubelet/pods/ddd42e3d-c494-4217-b211-09954d68f58e/volumes"
Oct 30 13:01:29.578479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:29.578423    4993 patch_prober.go:29] interesting pod/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:01:29.578978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:29.578513    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:30.253903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:30.253864    4993 generic.go:356] "Generic (PLEG): container finished" podID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerID="05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2" exitCode=0
Oct 30 13:01:30.254173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:30.253931    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" event={"ID":"1aa47a74-1ead-4407-8fab-2a84bf8f9172","Type":"ContainerDied","Data":"05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2"}
Oct 30 13:01:30.254516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:30.254496    4993 scope.go:122] "RemoveContainer" containerID="05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2"
Oct 30 13:01:30.256197 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:30.256169    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:01:30.256320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:30.256277    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:01:30.256320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:30.256315    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:01:31.290753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:31.290710    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" event={"ID":"1aa47a74-1ead-4407-8fab-2a84bf8f9172","Type":"ContainerStarted","Data":"5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf"}
Oct 30 13:01:32.125984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:32.125940    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:01:32.126256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:32.126186    4993 patch_prober.go:29] interesting pod/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 13:01:32.126351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:32.126291    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 13:01:33.125658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:33.125619    4993 patch_prober.go:29] interesting pod/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 13:01:33.126250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:33.125705    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 13:01:34.126447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:34.126402    4993 patch_prober.go:29] interesting pod/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 13:01:34.127046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:34.126482    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 13:01:34.991330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:34.991279    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"]
Oct 30 13:01:34.991927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:34.991909    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d4ce7766-a60f-4f37-b03d-cd36ed6a83f0" containerName="revoke-keys"
Oct 30 13:01:34.991927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:34.991927    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ce7766-a60f-4f37-b03d-cd36ed6a83f0" containerName="revoke-keys"
Oct 30 13:01:34.992065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:34.992030    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ce7766-a60f-4f37-b03d-cd36ed6a83f0" containerName="revoke-keys"
Oct 30 13:01:34.995683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:34.995650    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.001658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.001623    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"]
Oct 30 13:01:35.023133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.023093    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.023354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.023299    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.023473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.023450    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vpr\" (UniqueName: \"kubernetes.io/projected/eced43dd-db0b-44a3-82cc-27b0ed63914e-kube-api-access-r8vpr\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.023688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.023670    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:01:35.023805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.023786    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:01:35.033310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.033281    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:35.033459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.033334    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:01:35.124511 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.124468    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vpr\" (UniqueName: \"kubernetes.io/projected/eced43dd-db0b-44a3-82cc-27b0ed63914e-kube-api-access-r8vpr\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.124737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.124572    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.124737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.124649    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.125317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.125283    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.125668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.125642    4993 patch_prober.go:29] interesting pod/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 13:01:35.125719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.125693    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 13:01:35.127262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.127237    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.164252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.161074    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vpr\" (UniqueName: \"kubernetes.io/projected/eced43dd-db0b-44a3-82cc-27b0ed63914e-kube-api-access-r8vpr\") pod \"maas-api-revoke-keys-e2e-api-b-f399fe-h5k87\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.312897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.312810    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:35.705419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:35.705395    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"]
Oct 30 13:01:36.125568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:36.125526    4993 patch_prober.go:29] interesting pod/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 13:01:36.125803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:36.125594    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 13:01:36.433946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:36.433841    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87" event={"ID":"eced43dd-db0b-44a3-82cc-27b0ed63914e","Type":"ContainerStarted","Data":"6dc71a4cebc6c0e462bd610ea92b881415419cb47155741566e274a8ed8f5f25"}
Oct 30 13:01:36.433946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:36.433907    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87" event={"ID":"eced43dd-db0b-44a3-82cc-27b0ed63914e","Type":"ContainerStarted","Data":"1f4450a6da7662b4ce9f54897859cda74612673fc33d94b7e2be843a17c51eb0"}
Oct 30 13:01:37.144563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:37.144461    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:01:37.145034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:37.145010    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:01:37.147657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:37.147630    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:01:37.206311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:37.206245    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87" podStartSLOduration=3.206224234 podStartE2EDuration="3.206224234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:36.489573413 +0000 UTC m=+1760.965293021" watchObservedRunningTime="2026-10-30 13:01:37.206224234 +0000 UTC m=+1761.681943875"
Oct 30 13:01:37.445553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:37.445456    4993 generic.go:356] "Generic (PLEG): container finished" podID="eced43dd-db0b-44a3-82cc-27b0ed63914e" containerID="6dc71a4cebc6c0e462bd610ea92b881415419cb47155741566e274a8ed8f5f25" exitCode=0
Oct 30 13:01:37.445948 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:37.445551    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87" event={"ID":"eced43dd-db0b-44a3-82cc-27b0ed63914e","Type":"ContainerDied","Data":"6dc71a4cebc6c0e462bd610ea92b881415419cb47155741566e274a8ed8f5f25"}
Oct 30 13:01:38.580931 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:01:38.580887    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:01:38.636589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.636558    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:38.676624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.676574    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-openshift-ca\") pod \"eced43dd-db0b-44a3-82cc-27b0ed63914e\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") "
Oct 30 13:01:38.676624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.676616    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-xks-ca\") pod \"eced43dd-db0b-44a3-82cc-27b0ed63914e\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") "
Oct 30 13:01:38.676909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.676756    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eced43dd-db0b-44a3-82cc-27b0ed63914e-kube-api-access-r8vpr\" (UniqueName: \"kubernetes.io/projected/eced43dd-db0b-44a3-82cc-27b0ed63914e-kube-api-access-r8vpr\") pod \"eced43dd-db0b-44a3-82cc-27b0ed63914e\" (UID: \"eced43dd-db0b-44a3-82cc-27b0ed63914e\") "
Oct 30 13:01:38.677158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.677101    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-openshift-ca" pod "eced43dd-db0b-44a3-82cc-27b0ed63914e" (UID: "eced43dd-db0b-44a3-82cc-27b0ed63914e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:38.677571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.677541    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:38.680615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.680520    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-xks-ca" pod "eced43dd-db0b-44a3-82cc-27b0ed63914e" (UID: "eced43dd-db0b-44a3-82cc-27b0ed63914e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:38.680886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.680860    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eced43dd-db0b-44a3-82cc-27b0ed63914e-kube-api-access-r8vpr" pod "eced43dd-db0b-44a3-82cc-27b0ed63914e" (UID: "eced43dd-db0b-44a3-82cc-27b0ed63914e"). InnerVolumeSpecName "kube-api-access-r8vpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:38.781040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.780988    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eced43dd-db0b-44a3-82cc-27b0ed63914e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:38.781040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.781039    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vpr\" (UniqueName: \"kubernetes.io/projected/eced43dd-db0b-44a3-82cc-27b0ed63914e-kube-api-access-r8vpr\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:38.791105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.791064    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"]
Oct 30 13:01:38.791774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.791754    4993 cpu_manager.go:414] "containerMap: removing container" podUID="eced43dd-db0b-44a3-82cc-27b0ed63914e" containerName="revoke-keys"
Oct 30 13:01:38.791827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.791776    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eced43dd-db0b-44a3-82cc-27b0ed63914e" containerName="revoke-keys"
Oct 30 13:01:38.791901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.791892    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="eced43dd-db0b-44a3-82cc-27b0ed63914e" containerName="revoke-keys"
Oct 30 13:01:38.794720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.794701    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:38.804963 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.804928    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"]
Oct 30 13:01:38.882189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.882143    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:38.882422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.882341    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8t69\" (UniqueName: \"kubernetes.io/projected/49906f7f-81a7-4346-8130-d9b34b9c58c0-kube-api-access-k8t69\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:38.882469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.882423    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:38.984411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.984362    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:38.984589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.984568    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:38.984684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.984669    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8t69\" (UniqueName: \"kubernetes.io/projected/49906f7f-81a7-4346-8130-d9b34b9c58c0-kube-api-access-k8t69\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:38.985194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.985165    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:38.987330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:38.987301    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:39.008357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:39.008324    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8t69\" (UniqueName: \"kubernetes.io/projected/49906f7f-81a7-4346-8130-d9b34b9c58c0-kube-api-access-k8t69\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:39.112633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:39.112523    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:39.427746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:39.427711    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"]
Oct 30 13:01:39.427746 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:01:39.427728    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49906f7f_81a7_4346_8130_d9b34b9c58c0.slice/crio-2ee94f95032c32d6fb686455a01213de8017389203bd19b0a2d3a34c360126ea WatchSource:0}: Error finding container 2ee94f95032c32d6fb686455a01213de8017389203bd19b0a2d3a34c360126ea: Status 404 returned error can't find the container with id 2ee94f95032c32d6fb686455a01213de8017389203bd19b0a2d3a34c360126ea
Oct 30 13:01:39.472630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:39.472572    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9" event={"ID":"49906f7f-81a7-4346-8130-d9b34b9c58c0","Type":"ContainerStarted","Data":"2ee94f95032c32d6fb686455a01213de8017389203bd19b0a2d3a34c360126ea"}
Oct 30 13:01:39.474982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:39.474439    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87" event={"ID":"eced43dd-db0b-44a3-82cc-27b0ed63914e","Type":"ContainerDied","Data":"1f4450a6da7662b4ce9f54897859cda74612673fc33d94b7e2be843a17c51eb0"}
Oct 30 13:01:39.474982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:39.474508    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f4450a6da7662b4ce9f54897859cda74612673fc33d94b7e2be843a17c51eb0"
Oct 30 13:01:39.475305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:39.474991    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"
Oct 30 13:01:40.486966 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:40.486931    4993 generic.go:356] "Generic (PLEG): container finished" podID="49906f7f-81a7-4346-8130-d9b34b9c58c0" containerID="29247c441f0772718306ab97dc85d5142d7830a805e16d032fbec10ce7dbbe92" exitCode=0
Oct 30 13:01:40.487455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:40.487025    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9" event={"ID":"49906f7f-81a7-4346-8130-d9b34b9c58c0","Type":"ContainerDied","Data":"29247c441f0772718306ab97dc85d5142d7830a805e16d032fbec10ce7dbbe92"}
Oct 30 13:01:41.688125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.688096    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:41.723249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.723170    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49906f7f-81a7-4346-8130-d9b34b9c58c0-kube-api-access-k8t69\" (UniqueName: \"kubernetes.io/projected/49906f7f-81a7-4346-8130-d9b34b9c58c0-kube-api-access-k8t69\") pod \"49906f7f-81a7-4346-8130-d9b34b9c58c0\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") "
Oct 30 13:01:41.723470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.723451    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-openshift-ca\") pod \"49906f7f-81a7-4346-8130-d9b34b9c58c0\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") "
Oct 30 13:01:41.723535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.723505    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-xks-ca\") pod \"49906f7f-81a7-4346-8130-d9b34b9c58c0\" (UID: \"49906f7f-81a7-4346-8130-d9b34b9c58c0\") "
Oct 30 13:01:41.723927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.723896    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-openshift-ca" pod "49906f7f-81a7-4346-8130-d9b34b9c58c0" (UID: "49906f7f-81a7-4346-8130-d9b34b9c58c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:41.724607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.724583    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:41.725875 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.725851    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49906f7f-81a7-4346-8130-d9b34b9c58c0-kube-api-access-k8t69" pod "49906f7f-81a7-4346-8130-d9b34b9c58c0" (UID: "49906f7f-81a7-4346-8130-d9b34b9c58c0"). InnerVolumeSpecName "kube-api-access-k8t69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:41.725996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.725983    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-xks-ca" pod "49906f7f-81a7-4346-8130-d9b34b9c58c0" (UID: "49906f7f-81a7-4346-8130-d9b34b9c58c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:41.825557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.825459    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8t69\" (UniqueName: \"kubernetes.io/projected/49906f7f-81a7-4346-8130-d9b34b9c58c0-kube-api-access-k8t69\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:41.825557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:41.825495    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49906f7f-81a7-4346-8130-d9b34b9c58c0-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:42.508102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:42.508070    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"
Oct 30 13:01:42.508102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:42.508088    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9" event={"ID":"49906f7f-81a7-4346-8130-d9b34b9c58c0","Type":"ContainerDied","Data":"2ee94f95032c32d6fb686455a01213de8017389203bd19b0a2d3a34c360126ea"}
Oct 30 13:01:42.508358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:42.508133    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee94f95032c32d6fb686455a01213de8017389203bd19b0a2d3a34c360126ea"
Oct 30 13:01:43.463313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.463260    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"]
Oct 30 13:01:43.464190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.464169    4993 cpu_manager.go:414] "containerMap: removing container" podUID="49906f7f-81a7-4346-8130-d9b34b9c58c0" containerName="revoke-keys"
Oct 30 13:01:43.464267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.464194    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49906f7f-81a7-4346-8130-d9b34b9c58c0" containerName="revoke-keys"
Oct 30 13:01:43.464438 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.464424    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="49906f7f-81a7-4346-8130-d9b34b9c58c0" containerName="revoke-keys"
Oct 30 13:01:43.467821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.467795    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.483681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.483491    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:43.489720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.489679    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"]
Oct 30 13:01:43.500991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.495372    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:01:43.500991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.495538    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:01:43.500991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.495680    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:01:43.550666 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.550623    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.550864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.550836    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftjp\" (UniqueName: \"kubernetes.io/projected/cb2c86e9-738a-4873-9325-e7d3503676d2-kube-api-access-4ftjp\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.550922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.550877    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.652667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.652616    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftjp\" (UniqueName: \"kubernetes.io/projected/cb2c86e9-738a-4873-9325-e7d3503676d2-kube-api-access-4ftjp\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.652852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.652680    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.652852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.652825    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.653632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.653599    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.655432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.655400    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.679008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.678966    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftjp\" (UniqueName: \"kubernetes.io/projected/cb2c86e9-738a-4873-9325-e7d3503676d2-kube-api-access-4ftjp\") pod \"maas-api-revoke-sub-df169d4d-9m22c\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:43.781467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:43.781426    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:44.092336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:44.092297    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"]
Oct 30 13:01:44.092589 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:01:44.092557    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2c86e9_738a_4873_9325_e7d3503676d2.slice/crio-c17d14e2c87c3b2494c14cddc7528a178a2c8d26cfea6b629d199842256533ab WatchSource:0}: Error finding container c17d14e2c87c3b2494c14cddc7528a178a2c8d26cfea6b629d199842256533ab: Status 404 returned error can't find the container with id c17d14e2c87c3b2494c14cddc7528a178a2c8d26cfea6b629d199842256533ab
Oct 30 13:01:44.528761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:44.528726    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c" event={"ID":"cb2c86e9-738a-4873-9325-e7d3503676d2","Type":"ContainerStarted","Data":"8c75a4b88303a3ac72e869061b29915dfed77a3056d2df6c2c1b05c42bb50cd6"}
Oct 30 13:01:44.529172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:44.528784    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c" event={"ID":"cb2c86e9-738a-4873-9325-e7d3503676d2","Type":"ContainerStarted","Data":"c17d14e2c87c3b2494c14cddc7528a178a2c8d26cfea6b629d199842256533ab"}
Oct 30 13:01:44.572612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:44.572548    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c" podStartSLOduration=1.572532363 podStartE2EDuration="1.572532363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:44.569174609 +0000 UTC m=+1769.044894221" watchObservedRunningTime="2026-10-30 13:01:44.572532363 +0000 UTC m=+1769.048251973"
Oct 30 13:01:45.587713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:45.587669    4993 generic.go:356] "Generic (PLEG): container finished" podID="cb2c86e9-738a-4873-9325-e7d3503676d2" containerID="8c75a4b88303a3ac72e869061b29915dfed77a3056d2df6c2c1b05c42bb50cd6" exitCode=0
Oct 30 13:01:45.588250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:45.587744    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c" event={"ID":"cb2c86e9-738a-4873-9325-e7d3503676d2","Type":"ContainerDied","Data":"8c75a4b88303a3ac72e869061b29915dfed77a3056d2df6c2c1b05c42bb50cd6"}
Oct 30 13:01:46.785594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:46.785559    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:46.924117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:46.924008    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-xks-ca\") pod \"cb2c86e9-738a-4873-9325-e7d3503676d2\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") "
Oct 30 13:01:46.924117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:46.924062    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2c86e9-738a-4873-9325-e7d3503676d2-kube-api-access-4ftjp\" (UniqueName: \"kubernetes.io/projected/cb2c86e9-738a-4873-9325-e7d3503676d2-kube-api-access-4ftjp\") pod \"cb2c86e9-738a-4873-9325-e7d3503676d2\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") "
Oct 30 13:01:46.924536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:46.924268    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-openshift-ca\") pod \"cb2c86e9-738a-4873-9325-e7d3503676d2\" (UID: \"cb2c86e9-738a-4873-9325-e7d3503676d2\") "
Oct 30 13:01:46.924697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:46.924668    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-openshift-ca" pod "cb2c86e9-738a-4873-9325-e7d3503676d2" (UID: "cb2c86e9-738a-4873-9325-e7d3503676d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:46.924994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:46.924967    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:46.926451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:46.926413    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-xks-ca" pod "cb2c86e9-738a-4873-9325-e7d3503676d2" (UID: "cb2c86e9-738a-4873-9325-e7d3503676d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:46.926571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:46.926544    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2c86e9-738a-4873-9325-e7d3503676d2-kube-api-access-4ftjp" pod "cb2c86e9-738a-4873-9325-e7d3503676d2" (UID: "cb2c86e9-738a-4873-9325-e7d3503676d2"). InnerVolumeSpecName "kube-api-access-4ftjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:47.031401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:47.031321    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c86e9-738a-4873-9325-e7d3503676d2-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:47.031401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:47.031369    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ftjp\" (UniqueName: \"kubernetes.io/projected/cb2c86e9-738a-4873-9325-e7d3503676d2-kube-api-access-4ftjp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:47.612613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:47.612577    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"
Oct 30 13:01:47.612613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:47.612603    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c" event={"ID":"cb2c86e9-738a-4873-9325-e7d3503676d2","Type":"ContainerDied","Data":"c17d14e2c87c3b2494c14cddc7528a178a2c8d26cfea6b629d199842256533ab"}
Oct 30 13:01:47.612835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:47.612662    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17d14e2c87c3b2494c14cddc7528a178a2c8d26cfea6b629d199842256533ab"
Oct 30 13:01:49.922007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.921966    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"]
Oct 30 13:01:49.922611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.922593    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cb2c86e9-738a-4873-9325-e7d3503676d2" containerName="revoke-keys"
Oct 30 13:01:49.922673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.922612    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2c86e9-738a-4873-9325-e7d3503676d2" containerName="revoke-keys"
Oct 30 13:01:49.922741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.922731    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2c86e9-738a-4873-9325-e7d3503676d2" containerName="revoke-keys"
Oct 30 13:01:49.925504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.925484    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:49.946628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.946585    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"]
Oct 30 13:01:49.990231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.952260    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:49.990231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.952407    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:01:49.990231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.952526    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:01:49.990231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:49.952616    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:01:50.107300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.105489    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.107300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.105666    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntf6\" (UniqueName: \"kubernetes.io/projected/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-kube-api-access-qntf6\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.107300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.105866    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.210068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.209973    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.210068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.210065    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntf6\" (UniqueName: \"kubernetes.io/projected/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-kube-api-access-qntf6\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.210301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.210248    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.210870 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.210846    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.212520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.212496    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.238315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.238274    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntf6\" (UniqueName: \"kubernetes.io/projected/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-kube-api-access-qntf6\") pod \"maas-api-revoke-sub-5066e7be-mqtkx\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.239026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.239009    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:50.556105 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:01:50.556065    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8987d74_4eaf_4b4b_9028_9a13ea8c6dfc.slice/crio-46497fa4e3b0fa5380fb823f295b6c1aee02666b07712ec75b0f16c347e02193 WatchSource:0}: Error finding container 46497fa4e3b0fa5380fb823f295b6c1aee02666b07712ec75b0f16c347e02193: Status 404 returned error can't find the container with id 46497fa4e3b0fa5380fb823f295b6c1aee02666b07712ec75b0f16c347e02193
Oct 30 13:01:50.558566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.558532    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"]
Oct 30 13:01:50.638290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:50.638235    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx" event={"ID":"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc","Type":"ContainerStarted","Data":"46497fa4e3b0fa5380fb823f295b6c1aee02666b07712ec75b0f16c347e02193"}
Oct 30 13:01:51.648198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:51.648160    4993 generic.go:356] "Generic (PLEG): container finished" podID="e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc" containerID="70177dc58ab4151d643f03bc7341faf3e6dd07c911e1d57a4d69c2ae873c85ad" exitCode=0
Oct 30 13:01:51.648610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:51.648235    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx" event={"ID":"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc","Type":"ContainerDied","Data":"70177dc58ab4151d643f03bc7341faf3e6dd07c911e1d57a4d69c2ae873c85ad"}
Oct 30 13:01:52.827695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:52.827666    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:52.949996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:52.949908    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-xks-ca\") pod \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") "
Oct 30 13:01:52.950164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:52.950089    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-kube-api-access-qntf6\" (UniqueName: \"kubernetes.io/projected/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-kube-api-access-qntf6\") pod \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") "
Oct 30 13:01:52.950164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:52.950142    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-openshift-ca\") pod \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\" (UID: \"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc\") "
Oct 30 13:01:52.950514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:52.950487    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-openshift-ca" pod "e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc" (UID: "e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:52.950852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:52.950830    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:52.952116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:52.952093    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-xks-ca" pod "e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc" (UID: "e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:52.952296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:52.952273    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-kube-api-access-qntf6" pod "e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc" (UID: "e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc"). InnerVolumeSpecName "kube-api-access-qntf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:53.051988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:53.051954    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:53.051988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:53.051984    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntf6\" (UniqueName: \"kubernetes.io/projected/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc-kube-api-access-qntf6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:01:53.670760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:53.670727    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"
Oct 30 13:01:53.670760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:53.670746    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx" event={"ID":"e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc","Type":"ContainerDied","Data":"46497fa4e3b0fa5380fb823f295b6c1aee02666b07712ec75b0f16c347e02193"}
Oct 30 13:01:53.671038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:53.670804    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46497fa4e3b0fa5380fb823f295b6c1aee02666b07712ec75b0f16c347e02193"
Oct 30 13:01:57.132015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:57.131967    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"]
Oct 30 13:01:57.143593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:57.143554    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2e0e28-hr86v"]
Oct 30 13:01:57.693974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:01:57.693940    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ce7766-a60f-4f37-b03d-cd36ed6a83f0" path="/var/lib/kubelet/pods/d4ce7766-a60f-4f37-b03d-cd36ed6a83f0/volumes"
Oct 30 13:02:05.512871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.512821    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"]
Oct 30 13:02:05.513699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.513675    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc" containerName="revoke-keys"
Oct 30 13:02:05.513802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.513700    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc" containerName="revoke-keys"
Oct 30 13:02:05.513881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.513868    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc" containerName="revoke-keys"
Oct 30 13:02:05.517505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.517481    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.538878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.538840    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"]
Oct 30 13:02:05.545469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.545384    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:02:05.545469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.545373    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:05.545469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.545428    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:02:05.545760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.545547    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:02:05.586416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.586374    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.586635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.586515    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znc82\" (UniqueName: \"kubernetes.io/projected/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-kube-api-access-znc82\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.586635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.586573    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.691678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.691629    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.691965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.691764    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znc82\" (UniqueName: \"kubernetes.io/projected/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-kube-api-access-znc82\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.691965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.691838    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.694196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.693977    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.697430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.697398    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.718889 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.718851    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znc82\" (UniqueName: \"kubernetes.io/projected/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-kube-api-access-znc82\") pod \"maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:05.830426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:05.830336    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:06.156593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:06.156553    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"]
Oct 30 13:02:06.157035 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:02:06.157007    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef914291_1fd8_4c0c_82ec_080a9bdff4fc.slice/crio-856af2c085e1a6b795dba6824792b6bb17c380eae33a5fbf31f95dbcba8b48fd WatchSource:0}: Error finding container 856af2c085e1a6b795dba6824792b6bb17c380eae33a5fbf31f95dbcba8b48fd: Status 404 returned error can't find the container with id 856af2c085e1a6b795dba6824792b6bb17c380eae33a5fbf31f95dbcba8b48fd
Oct 30 13:02:06.833437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:06.833401    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6" event={"ID":"ef914291-1fd8-4c0c-82ec-080a9bdff4fc","Type":"ContainerStarted","Data":"e3b3c67b9b6023622081fdf08a94f9fd3406b5db4790786c14a10138563e081b"}
Oct 30 13:02:06.833875 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:06.833457    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6" event={"ID":"ef914291-1fd8-4c0c-82ec-080a9bdff4fc","Type":"ContainerStarted","Data":"856af2c085e1a6b795dba6824792b6bb17c380eae33a5fbf31f95dbcba8b48fd"}
Oct 30 13:02:06.876402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:06.876335    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6" podStartSLOduration=1.876319085 podStartE2EDuration="1.876319085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:06.873677465 +0000 UTC m=+1791.349397099" watchObservedRunningTime="2026-10-30 13:02:06.876319085 +0000 UTC m=+1791.352038694"
Oct 30 13:02:07.731122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:07.731082    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"]
Oct 30 13:02:07.743307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:07.743265    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8rgb9"]
Oct 30 13:02:07.846726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:07.846684    4993 generic.go:356] "Generic (PLEG): container finished" podID="ef914291-1fd8-4c0c-82ec-080a9bdff4fc" containerID="e3b3c67b9b6023622081fdf08a94f9fd3406b5db4790786c14a10138563e081b" exitCode=0
Oct 30 13:02:07.847188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:07.846768    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6" event={"ID":"ef914291-1fd8-4c0c-82ec-080a9bdff4fc","Type":"ContainerDied","Data":"e3b3c67b9b6023622081fdf08a94f9fd3406b5db4790786c14a10138563e081b"}
Oct 30 13:02:08.570674 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:02:08.570632    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:02:09.021988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.021960    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:09.190902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.190787    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-kube-api-access-znc82\" (UniqueName: \"kubernetes.io/projected/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-kube-api-access-znc82\") pod \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") "
Oct 30 13:02:09.191092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.191012    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-openshift-ca\") pod \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") "
Oct 30 13:02:09.191092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.191044    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-xks-ca\") pod \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\" (UID: \"ef914291-1fd8-4c0c-82ec-080a9bdff4fc\") "
Oct 30 13:02:09.191786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.191757    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-openshift-ca" pod "ef914291-1fd8-4c0c-82ec-080a9bdff4fc" (UID: "ef914291-1fd8-4c0c-82ec-080a9bdff4fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:09.193731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.193701    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-kube-api-access-znc82" pod "ef914291-1fd8-4c0c-82ec-080a9bdff4fc" (UID: "ef914291-1fd8-4c0c-82ec-080a9bdff4fc"). InnerVolumeSpecName "kube-api-access-znc82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:09.194007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.193979    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-xks-ca" pod "ef914291-1fd8-4c0c-82ec-080a9bdff4fc" (UID: "ef914291-1fd8-4c0c-82ec-080a9bdff4fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:09.294252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.293123    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znc82\" (UniqueName: \"kubernetes.io/projected/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-kube-api-access-znc82\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.294252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.293164    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.294252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.293181    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef914291-1fd8-4c0c-82ec-080a9bdff4fc-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.718227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.716404    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49906f7f-81a7-4346-8130-d9b34b9c58c0" path="/var/lib/kubelet/pods/49906f7f-81a7-4346-8130-d9b34b9c58c0/volumes"
Oct 30 13:02:09.833327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.833287    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"]
Oct 30 13:02:09.845555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.845511    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f399fe-h5k87"]
Oct 30 13:02:09.868298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.868263    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"
Oct 30 13:02:09.868483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.868263    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6" event={"ID":"ef914291-1fd8-4c0c-82ec-080a9bdff4fc","Type":"ContainerDied","Data":"856af2c085e1a6b795dba6824792b6bb17c380eae33a5fbf31f95dbcba8b48fd"}
Oct 30 13:02:09.868483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:09.868391    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="856af2c085e1a6b795dba6824792b6bb17c380eae33a5fbf31f95dbcba8b48fd"
Oct 30 13:02:11.694289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:11.694249    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eced43dd-db0b-44a3-82cc-27b0ed63914e" path="/var/lib/kubelet/pods/eced43dd-db0b-44a3-82cc-27b0ed63914e/volumes"
Oct 30 13:02:15.939599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:15.939561    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:02:15.948246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:15.948201    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:02:16.915417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:16.915380    4993 scope.go:122] "RemoveContainer" containerID="119297ee4d7e3b964807be47b2f4aafd9c3b2f027d84ea6945ba50f452db361d"
Oct 30 13:02:16.944116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:16.944085    4993 scope.go:122] "RemoveContainer" containerID="1bcf142e8bed2460ccc50a534653e91af1f09bb531a8fcd78a47b5350f749dcf"
Oct 30 13:02:16.969541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:16.969393    4993 scope.go:122] "RemoveContainer" containerID="a1ba9b86667366a6fdbd42108f799efc3c22e90c650b900eab0abdab932ef894"
Oct 30 13:02:16.986995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:16.986968    4993 scope.go:122] "RemoveContainer" containerID="9140b26eed58b0d9e2c7722b78de45fb62509d2d2058da94a04993a046b359a7"
Oct 30 13:02:17.001330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:17.001269    4993 scope.go:122] "RemoveContainer" containerID="39476e280fb356178ae87d7d023f1494589e0e5996a3816e583d1f0a2fda4e45"
Oct 30 13:02:17.012455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:17.012425    4993 scope.go:122] "RemoveContainer" containerID="a3344f60f8d6a56e1f306b13b1e4d3470cc328ff5a4e4d08ece688552e97f755"
Oct 30 13:02:17.023709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:17.023684    4993 scope.go:122] "RemoveContainer" containerID="a4d3abc93a27db8296664ecd27d09917fb63beccc27490c570289aeb466ee26d"
Oct 30 13:02:32.044228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:32.044163    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"]
Oct 30 13:02:32.057920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:32.057869    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318ba379-pkhzb"]
Oct 30 13:02:33.697618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:33.697570    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5546197-1610-45c0-9227-dcdbc6d2c2bc" path="/var/lib/kubelet/pods/f5546197-1610-45c0-9227-dcdbc6d2c2bc/volumes"
Oct 30 13:02:38.570681 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:02:38.570634    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:02:43.033307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:43.033271    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"]
Oct 30 13:02:43.046569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:43.046536    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16fdbbb-4v54z"]
Oct 30 13:02:43.745231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:43.735780    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795cf42f-68d3-46bf-ac6e-90ce64f564aa" path="/var/lib/kubelet/pods/795cf42f-68d3-46bf-ac6e-90ce64f564aa/volumes"
Oct 30 13:02:44.044624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:44.044523    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"]
Oct 30 13:02:44.058481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:44.058433    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd05489-cqdk5"]
Oct 30 13:02:45.696358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:45.696314    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5ea238-7888-48d6-94d0-21c4216e43a5" path="/var/lib/kubelet/pods/1c5ea238-7888-48d6-94d0-21c4216e43a5/volumes"
Oct 30 13:02:47.166896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.166856    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"]
Oct 30 13:02:47.167838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.167813    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ef914291-1fd8-4c0c-82ec-080a9bdff4fc" containerName="revoke-keys"
Oct 30 13:02:47.167838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.167838    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef914291-1fd8-4c0c-82ec-080a9bdff4fc" containerName="revoke-keys"
Oct 30 13:02:47.168037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.167985    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef914291-1fd8-4c0c-82ec-080a9bdff4fc" containerName="revoke-keys"
Oct 30 13:02:47.173462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.173436    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.194078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.193289    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:02:47.208299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.204677    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:47.208299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.205052    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:02:47.208299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.208036    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:02:47.208299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.208036    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.208299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.208169    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68z5j\" (UniqueName: \"kubernetes.io/projected/1c75f50f-92e4-4f78-85d1-5809b21d3839-kube-api-access-68z5j\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.208696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.208366    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.233750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.233708    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"]
Oct 30 13:02:47.310614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.310570    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.310804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.310632    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68z5j\" (UniqueName: \"kubernetes.io/projected/1c75f50f-92e4-4f78-85d1-5809b21d3839-kube-api-access-68z5j\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.310804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.310781    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.311381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.311354    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.313232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.313189    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.334674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.334640    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68z5j\" (UniqueName: \"kubernetes.io/projected/1c75f50f-92e4-4f78-85d1-5809b21d3839-kube-api-access-68z5j\") pod \"maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.537091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.537040    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:47.855001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:47.854966    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"]
Oct 30 13:02:48.039327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:48.039280    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"]
Oct 30 13:02:48.044340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:48.044305    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42219dde-8pddb"]
Oct 30 13:02:48.336262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:48.336184    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq" event={"ID":"1c75f50f-92e4-4f78-85d1-5809b21d3839","Type":"ContainerStarted","Data":"2299aa33da0cd9d7dd6697ceb3aefa0b9c80f0dafa4fc48b8116209665225b58"}
Oct 30 13:02:48.336688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:48.336285    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq" event={"ID":"1c75f50f-92e4-4f78-85d1-5809b21d3839","Type":"ContainerStarted","Data":"258c7d9e02d633a518576945d37b58d2f39845662dc9b52727f778f06aea54b9"}
Oct 30 13:02:48.386874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:48.386804    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq" podStartSLOduration=1.386783129 podStartE2EDuration="1.386783129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:48.386490987 +0000 UTC m=+1832.862210599" watchObservedRunningTime="2026-10-30 13:02:48.386783129 +0000 UTC m=+1832.862502741"
Oct 30 13:02:49.346154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:49.346119    4993 generic.go:356] "Generic (PLEG): container finished" podID="1c75f50f-92e4-4f78-85d1-5809b21d3839" containerID="2299aa33da0cd9d7dd6697ceb3aefa0b9c80f0dafa4fc48b8116209665225b58" exitCode=0
Oct 30 13:02:49.346588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:49.346202    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq" event={"ID":"1c75f50f-92e4-4f78-85d1-5809b21d3839","Type":"ContainerDied","Data":"2299aa33da0cd9d7dd6697ceb3aefa0b9c80f0dafa4fc48b8116209665225b58"}
Oct 30 13:02:49.693567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:49.693487    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd506a1-f907-4021-a981-9c5723bc38c9" path="/var/lib/kubelet/pods/bdd506a1-f907-4021-a981-9c5723bc38c9/volumes"
Oct 30 13:02:50.681465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.681427    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:50.806720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.806671    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-xks-ca\") pod \"1c75f50f-92e4-4f78-85d1-5809b21d3839\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") "
Oct 30 13:02:50.852777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.845860    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-openshift-ca\") pod \"1c75f50f-92e4-4f78-85d1-5809b21d3839\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") "
Oct 30 13:02:50.853607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.853580    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c75f50f-92e4-4f78-85d1-5809b21d3839-kube-api-access-68z5j\" (UniqueName: \"kubernetes.io/projected/1c75f50f-92e4-4f78-85d1-5809b21d3839-kube-api-access-68z5j\") pod \"1c75f50f-92e4-4f78-85d1-5809b21d3839\" (UID: \"1c75f50f-92e4-4f78-85d1-5809b21d3839\") "
Oct 30 13:02:50.854394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.847726    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-openshift-ca" pod "1c75f50f-92e4-4f78-85d1-5809b21d3839" (UID: "1c75f50f-92e4-4f78-85d1-5809b21d3839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:50.854543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.853377    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-xks-ca" pod "1c75f50f-92e4-4f78-85d1-5809b21d3839" (UID: "1c75f50f-92e4-4f78-85d1-5809b21d3839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:50.855299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.855278    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:02:50.855428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.855396    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c75f50f-92e4-4f78-85d1-5809b21d3839-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:02:50.856466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.856432    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c75f50f-92e4-4f78-85d1-5809b21d3839-kube-api-access-68z5j" pod "1c75f50f-92e4-4f78-85d1-5809b21d3839" (UID: "1c75f50f-92e4-4f78-85d1-5809b21d3839"). InnerVolumeSpecName "kube-api-access-68z5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:50.956533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:50.956447    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68z5j\" (UniqueName: \"kubernetes.io/projected/1c75f50f-92e4-4f78-85d1-5809b21d3839-kube-api-access-68z5j\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:02:51.017759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:51.017709    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"]
Oct 30 13:02:51.028639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:51.028600    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4561d2-t4wf6"]
Oct 30 13:02:51.367846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:51.367815    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"
Oct 30 13:02:51.368035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:51.367847    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq" event={"ID":"1c75f50f-92e4-4f78-85d1-5809b21d3839","Type":"ContainerDied","Data":"258c7d9e02d633a518576945d37b58d2f39845662dc9b52727f778f06aea54b9"}
Oct 30 13:02:51.368035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:51.367895    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="258c7d9e02d633a518576945d37b58d2f39845662dc9b52727f778f06aea54b9"
Oct 30 13:02:51.692844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:51.692753    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef914291-1fd8-4c0c-82ec-080a9bdff4fc" path="/var/lib/kubelet/pods/ef914291-1fd8-4c0c-82ec-080a9bdff4fc/volumes"
Oct 30 13:02:53.052468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:53.052429    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"]
Oct 30 13:02:53.062845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:53.062812    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124ed98f-2m56r"]
Oct 30 13:02:53.692892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:53.692818    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8985b6-144d-4cae-b778-b32ddb1fd79a" path="/var/lib/kubelet/pods/2d8985b6-144d-4cae-b778-b32ddb1fd79a/volumes"
Oct 30 13:02:58.709098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:58.708114    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"]
Oct 30 13:02:58.709098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:02:58.708465    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy" containerID="cri-o://5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf" gracePeriod=30
Oct 30 13:03:04.020511 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.020480    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:03:04.176872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.176823    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-token\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-token\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.176882    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-socket\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.176978    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-kube-api-access-kftcj\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-kube-api-access-kftcj\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177007    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istiod-ca-cert\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177055    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-credential-socket\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177182    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-certs\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177218    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-socket" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:04.177311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177263    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-envoy\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177344    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-credential-socket" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:04.177550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177397    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-data\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177475    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-podinfo\") pod \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\" (UID: \"1aa47a74-1ead-4407-8fab-2a84bf8f9172\") "
Oct 30 13:03:04.177550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177493    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-certs" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:04.177550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177516    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istiod-ca-cert" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:04.177799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.177690    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-data" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:04.178092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.178069    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.178185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.178098    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.178185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.178114    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.178185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.178132    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.178185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.178146    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.179553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.179530    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-kube-api-access-kftcj" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "kube-api-access-kftcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:04.179672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.179581    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-token" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:04.180060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.180036    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-envoy" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:03:04.180177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.180155    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-podinfo" pod "1aa47a74-1ead-4407-8fab-2a84bf8f9172" (UID: "1aa47a74-1ead-4407-8fab-2a84bf8f9172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:03:04.279713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.279623    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.279713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.279660    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.279713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.279671    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.279713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.279682    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kftcj\" (UniqueName: \"kubernetes.io/projected/1aa47a74-1ead-4407-8fab-2a84bf8f9172-kube-api-access-kftcj\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:04.551426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.551314    4993 generic.go:356] "Generic (PLEG): container finished" podID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerID="5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf" exitCode=0
Oct 30 13:03:04.551426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.551386    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"
Oct 30 13:03:04.551673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.551410    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" event={"ID":"1aa47a74-1ead-4407-8fab-2a84bf8f9172","Type":"ContainerDied","Data":"5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf"}
Oct 30 13:03:04.551673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.551485    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2" event={"ID":"1aa47a74-1ead-4407-8fab-2a84bf8f9172","Type":"ContainerDied","Data":"3f3810ff558cd8a6e2a1caec50464520a9b59dd57f03ee6f3686f54632a2a2de"}
Oct 30 13:03:04.551673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.551518    4993 scope.go:122] "RemoveContainer" containerID="5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf"
Oct 30 13:03:04.569199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.568855    4993 scope.go:122] "RemoveContainer" containerID="05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2"
Oct 30 13:03:04.581291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.581193    4993 scope.go:122] "RemoveContainer" containerID="5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf"
Oct 30 13:03:04.581722 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:03:04.581699    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf\": container with ID starting with 5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf not found: ID does not exist" containerID="5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf"
Oct 30 13:03:04.581843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.581732    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf"} err="failed to get container status \"5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf\": rpc error: code = NotFound desc = could not find container \"5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf\": container with ID starting with 5e4f9b0841dae3e851a7e2d2a550760dc274f6e05f84a85f3bbc1801dd88c5cf not found: ID does not exist"
Oct 30 13:03:04.581843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.581752    4993 scope.go:122] "RemoveContainer" containerID="05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2"
Oct 30 13:03:04.582639 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:03:04.582610    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2\": container with ID starting with 05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2 not found: ID does not exist" containerID="05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2"
Oct 30 13:03:04.582792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.582767    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2"} err="failed to get container status \"05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2\": rpc error: code = NotFound desc = could not find container \"05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2\": container with ID starting with 05fd44283c9f04ae7d3443b9a2430fd0cfc7f10c5f31c862d63f564e52fb6ee2 not found: ID does not exist"
Oct 30 13:03:04.610636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.610583    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"]
Oct 30 13:03:04.616188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:04.616151    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4561d2-openshift-default-854f7c5995-87tt2"]
Oct 30 13:03:05.639034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:05.637651    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"]
Oct 30 13:03:05.644129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:05.644087    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea117f29-8v6xq"]
Oct 30 13:03:05.692631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:05.692598    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" path="/var/lib/kubelet/pods/1aa47a74-1ead-4407-8fab-2a84bf8f9172/volumes"
Oct 30 13:03:05.693072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:05.693058    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c75f50f-92e4-4f78-85d1-5809b21d3839" path="/var/lib/kubelet/pods/1c75f50f-92e4-4f78-85d1-5809b21d3839/volumes"
Oct 30 13:03:08.038394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:08.038354    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"]
Oct 30 13:03:08.050449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:08.050406    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc46f53-6jbzq"]
Oct 30 13:03:08.579632 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:03:08.579578    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:03:09.694292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:09.694254    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be38df56-d9bb-40c0-950e-5a9901eb2030" path="/var/lib/kubelet/pods/be38df56-d9bb-40c0-950e-5a9901eb2030/volumes"
Oct 30 13:03:12.031825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:12.031787    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"]
Oct 30 13:03:12.043084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:12.043048    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc132f0d-ll7tv"]
Oct 30 13:03:13.692426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:13.692383    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59396606-0545-4936-a10a-d5ca3cd99368" path="/var/lib/kubelet/pods/59396606-0545-4936-a10a-d5ca3cd99368/volumes"
Oct 30 13:03:15.936800 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.936763    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"]
Oct 30 13:03:15.937373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.937357    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy"
Oct 30 13:03:15.937422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.937375    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy"
Oct 30 13:03:15.937457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.937425    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1c75f50f-92e4-4f78-85d1-5809b21d3839" containerName="revoke-keys"
Oct 30 13:03:15.937457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.937431    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c75f50f-92e4-4f78-85d1-5809b21d3839" containerName="revoke-keys"
Oct 30 13:03:15.937537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.937526    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy"
Oct 30 13:03:15.937573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.937541    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c75f50f-92e4-4f78-85d1-5809b21d3839" containerName="revoke-keys"
Oct 30 13:03:15.940776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.940755    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:15.959504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.959474    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:03:15.970671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.970628    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:03:15.977758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.975421    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:03:15.977758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.975955    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:15.977758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:15.977682    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"]
Oct 30 13:03:16.057357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.057323    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.057566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.057368    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.057566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.057397    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6gm\" (UniqueName: \"kubernetes.io/projected/d7c45263-ad41-459f-a3ae-218cc22e0855-kube-api-access-4f6gm\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.159078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.159020    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.159078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.159087    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.159359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.159118    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6gm\" (UniqueName: \"kubernetes.io/projected/d7c45263-ad41-459f-a3ae-218cc22e0855-kube-api-access-4f6gm\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.159780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.159754    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.161617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.161588    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.182826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.182794    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6gm\" (UniqueName: \"kubernetes.io/projected/d7c45263-ad41-459f-a3ae-218cc22e0855-kube-api-access-4f6gm\") pod \"maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.254238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.254188    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:16.564392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.564290    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"]
Oct 30 13:03:16.564661 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:03:16.564626    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c45263_ad41_459f_a3ae_218cc22e0855.slice/crio-bf060a294c1192c3feada587854ea69b2586c5d8c23fd1ec2c6cca0c133d6f6a WatchSource:0}: Error finding container bf060a294c1192c3feada587854ea69b2586c5d8c23fd1ec2c6cca0c133d6f6a: Status 404 returned error can't find the container with id bf060a294c1192c3feada587854ea69b2586c5d8c23fd1ec2c6cca0c133d6f6a
Oct 30 13:03:16.676916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:16.676861    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c" event={"ID":"d7c45263-ad41-459f-a3ae-218cc22e0855","Type":"ContainerStarted","Data":"bf060a294c1192c3feada587854ea69b2586c5d8c23fd1ec2c6cca0c133d6f6a"}
Oct 30 13:03:17.687716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:17.687678    4993 generic.go:356] "Generic (PLEG): container finished" podID="d7c45263-ad41-459f-a3ae-218cc22e0855" containerID="f6da0bd4deeae888c7e7e2e0e3a4fd33a51c5d232d3842ba0b1c9b6c36d5266f" exitCode=0
Oct 30 13:03:17.691750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:17.691710    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c" event={"ID":"d7c45263-ad41-459f-a3ae-218cc22e0855","Type":"ContainerDied","Data":"f6da0bd4deeae888c7e7e2e0e3a4fd33a51c5d232d3842ba0b1c9b6c36d5266f"}
Oct 30 13:03:18.872737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:18.872701    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:18.999854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:18.999817    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-openshift-ca\") pod \"d7c45263-ad41-459f-a3ae-218cc22e0855\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") "
Oct 30 13:03:19.000033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:18.999910    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c45263-ad41-459f-a3ae-218cc22e0855-kube-api-access-4f6gm\" (UniqueName: \"kubernetes.io/projected/d7c45263-ad41-459f-a3ae-218cc22e0855-kube-api-access-4f6gm\") pod \"d7c45263-ad41-459f-a3ae-218cc22e0855\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") "
Oct 30 13:03:19.000033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.000000    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-xks-ca\") pod \"d7c45263-ad41-459f-a3ae-218cc22e0855\" (UID: \"d7c45263-ad41-459f-a3ae-218cc22e0855\") "
Oct 30 13:03:19.000411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.000381    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-openshift-ca" pod "d7c45263-ad41-459f-a3ae-218cc22e0855" (UID: "d7c45263-ad41-459f-a3ae-218cc22e0855"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:19.000618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.000599    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:19.002347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.002314    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-xks-ca" pod "d7c45263-ad41-459f-a3ae-218cc22e0855" (UID: "d7c45263-ad41-459f-a3ae-218cc22e0855"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:19.002464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.002389    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c45263-ad41-459f-a3ae-218cc22e0855-kube-api-access-4f6gm" pod "d7c45263-ad41-459f-a3ae-218cc22e0855" (UID: "d7c45263-ad41-459f-a3ae-218cc22e0855"). InnerVolumeSpecName "kube-api-access-4f6gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:19.102457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.102411    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f6gm\" (UniqueName: \"kubernetes.io/projected/d7c45263-ad41-459f-a3ae-218cc22e0855-kube-api-access-4f6gm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:19.102457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.102457    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c45263-ad41-459f-a3ae-218cc22e0855-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:03:19.709172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.709144    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"
Oct 30 13:03:19.709373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.709142    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c" event={"ID":"d7c45263-ad41-459f-a3ae-218cc22e0855","Type":"ContainerDied","Data":"bf060a294c1192c3feada587854ea69b2586c5d8c23fd1ec2c6cca0c133d6f6a"}
Oct 30 13:03:19.709373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:19.709268    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf060a294c1192c3feada587854ea69b2586c5d8c23fd1ec2c6cca0c133d6f6a"
Oct 30 13:03:21.033811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:21.033763    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"]
Oct 30 13:03:21.046128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:21.046083    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0e24fae-h26mg"]
Oct 30 13:03:21.694352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:21.694313    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76feb2a3-41e7-4718-8980-af66eb089179" path="/var/lib/kubelet/pods/76feb2a3-41e7-4718-8980-af66eb089179/volumes"
Oct 30 13:03:25.027616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:25.027573    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"]
Oct 30 13:03:25.039316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:25.039267    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9bc41ff-5wlsf"]
Oct 30 13:03:25.694632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:25.694589    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8586284e-14f9-4b07-955e-3054491dd750" path="/var/lib/kubelet/pods/8586284e-14f9-4b07-955e-3054491dd750/volumes"
Oct 30 13:03:31.041522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:31.041474    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"]
Oct 30 13:03:31.055879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:31.055844    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9239531-58glr"]
Oct 30 13:03:31.693945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:31.693901    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7dfbab-ae0f-417d-a935-f0b887dcb782" path="/var/lib/kubelet/pods/3f7dfbab-ae0f-417d-a935-f0b887dcb782/volumes"
Oct 30 13:03:36.065361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:36.065320    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"]
Oct 30 13:03:36.076972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:36.076925    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3085f50-x6dpc"]
Oct 30 13:03:37.695703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:37.695658    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5705f87-acc0-4c1d-8d65-7294d7d77b63" path="/var/lib/kubelet/pods/c5705f87-acc0-4c1d-8d65-7294d7d77b63/volumes"
Oct 30 13:03:38.572498 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:03:38.572451    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:03:46.037530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:46.037479    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"]
Oct 30 13:03:46.049072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:46.049030    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1316aad-llmpw"]
Oct 30 13:03:47.693635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:47.693594    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b9e9d7-d593-4abf-bac3-f72c72d69590" path="/var/lib/kubelet/pods/d9b9e9d7-d593-4abf-bac3-f72c72d69590/volumes"
Oct 30 13:03:48.685821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:48.685783    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"]
Oct 30 13:03:48.703592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:48.703540    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f60f16-4wf7c"]
Oct 30 13:03:49.698309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:49.698263    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c45263-ad41-459f-a3ae-218cc22e0855" path="/var/lib/kubelet/pods/d7c45263-ad41-459f-a3ae-218cc22e0855/volumes"
Oct 30 13:03:51.047647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:51.047611    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"]
Oct 30 13:03:51.058919 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:51.058878    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8a04de-k97gh"]
Oct 30 13:03:51.695690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:51.695646    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7e6fdf-d388-4793-aa93-5c7055c7ab07" path="/var/lib/kubelet/pods/cf7e6fdf-d388-4793-aa93-5c7055c7ab07/volumes"
Oct 30 13:03:56.878072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.878018    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"]
Oct 30 13:03:56.878920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.878897    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d7c45263-ad41-459f-a3ae-218cc22e0855" containerName="revoke-keys"
Oct 30 13:03:56.878967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.878924    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c45263-ad41-459f-a3ae-218cc22e0855" containerName="revoke-keys"
Oct 30 13:03:56.879002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.878994    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy"
Oct 30 13:03:56.879036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.879002    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy"
Oct 30 13:03:56.879154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.879142    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c45263-ad41-459f-a3ae-218cc22e0855" containerName="revoke-keys"
Oct 30 13:03:56.879196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.879159    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa47a74-1ead-4407-8fab-2a84bf8f9172" containerName="istio-proxy"
Oct 30 13:03:56.883494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.883465    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:56.899826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.899791    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:03:56.911335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.911298    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:03:56.921798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.921760    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"]
Oct 30 13:03:56.954227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.954177    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:56.954435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:56.954302    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:03:57.036925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.036880    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"]
Oct 30 13:03:57.041076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.041034    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-177c2301-fsw44"]
Oct 30 13:03:57.066196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.066151    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.066450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.066256    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.066450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.066417    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48l7x\" (UniqueName: \"kubernetes.io/projected/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-kube-api-access-48l7x\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.168003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.167911    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.168003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.167968    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.168287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.168006    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48l7x\" (UniqueName: \"kubernetes.io/projected/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-kube-api-access-48l7x\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.168653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.168628    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.170592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.170566    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.192677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.192647    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48l7x\" (UniqueName: \"kubernetes.io/projected/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-kube-api-access-48l7x\") pod \"maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.198577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.198547    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:03:57.514285 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:03:57.514252    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ba3b43_38f9_45c6_80c8_b4cb5e7f7209.slice/crio-2833e8cff59d93a8fe97f2d5e93cd7c974fd58cc3744b22fe2fb1e8d3ac3b1d5 WatchSource:0}: Error finding container 2833e8cff59d93a8fe97f2d5e93cd7c974fd58cc3744b22fe2fb1e8d3ac3b1d5: Status 404 returned error can't find the container with id 2833e8cff59d93a8fe97f2d5e93cd7c974fd58cc3744b22fe2fb1e8d3ac3b1d5
Oct 30 13:03:57.514998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.514971    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"]
Oct 30 13:03:57.692488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:57.692450    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c46fa7a-df92-4fa8-a054-9de3c0620cca" path="/var/lib/kubelet/pods/7c46fa7a-df92-4fa8-a054-9de3c0620cca/volumes"
Oct 30 13:03:58.115755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:58.115651    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt" event={"ID":"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209","Type":"ContainerStarted","Data":"c05a81a9d44eac3789db9854210d3b34003e2dd97760a448ccc62eb815c58867"}
Oct 30 13:03:58.115755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:58.115721    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt" event={"ID":"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209","Type":"ContainerStarted","Data":"2833e8cff59d93a8fe97f2d5e93cd7c974fd58cc3744b22fe2fb1e8d3ac3b1d5"}
Oct 30 13:03:58.166298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:58.166226    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt" podStartSLOduration=2.166187301 podStartE2EDuration="2.166187301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:58.160473083 +0000 UTC m=+1902.636192693" watchObservedRunningTime="2026-10-30 13:03:58.166187301 +0000 UTC m=+1902.641906912"
Oct 30 13:03:59.127906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:59.127867    4993 generic.go:356] "Generic (PLEG): container finished" podID="82ba3b43-38f9-45c6-80c8-b4cb5e7f7209" containerID="c05a81a9d44eac3789db9854210d3b34003e2dd97760a448ccc62eb815c58867" exitCode=0
Oct 30 13:03:59.128410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:03:59.127956    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt" event={"ID":"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209","Type":"ContainerDied","Data":"c05a81a9d44eac3789db9854210d3b34003e2dd97760a448ccc62eb815c58867"}
Oct 30 13:04:00.039584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.039532    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"]
Oct 30 13:04:00.048738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.048704    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbac848a-kr26t"]
Oct 30 13:04:00.331939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.331903    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:04:00.349860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.349825    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-openshift-ca\") pod \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") "
Oct 30 13:04:00.350040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.349907    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-kube-api-access-48l7x\" (UniqueName: \"kubernetes.io/projected/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-kube-api-access-48l7x\") pod \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") "
Oct 30 13:04:00.350040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.349944    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-xks-ca\") pod \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\" (UID: \"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209\") "
Oct 30 13:04:00.350295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.350264    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-openshift-ca" pod "82ba3b43-38f9-45c6-80c8-b4cb5e7f7209" (UID: "82ba3b43-38f9-45c6-80c8-b4cb5e7f7209"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:00.350661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.350640    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:00.352307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.352273    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-xks-ca" pod "82ba3b43-38f9-45c6-80c8-b4cb5e7f7209" (UID: "82ba3b43-38f9-45c6-80c8-b4cb5e7f7209"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:00.352419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.352395    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-kube-api-access-48l7x" pod "82ba3b43-38f9-45c6-80c8-b4cb5e7f7209" (UID: "82ba3b43-38f9-45c6-80c8-b4cb5e7f7209"). InnerVolumeSpecName "kube-api-access-48l7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:00.452276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.452193    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48l7x\" (UniqueName: \"kubernetes.io/projected/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-kube-api-access-48l7x\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:00.452276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.452268    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:00.520792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.520751    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"]
Oct 30 13:04:00.521724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.521695    4993 cpu_manager.go:414] "containerMap: removing container" podUID="82ba3b43-38f9-45c6-80c8-b4cb5e7f7209" containerName="revoke-keys"
Oct 30 13:04:00.521724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.521725    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ba3b43-38f9-45c6-80c8-b4cb5e7f7209" containerName="revoke-keys"
Oct 30 13:04:00.521956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.521937    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ba3b43-38f9-45c6-80c8-b4cb5e7f7209" containerName="revoke-keys"
Oct 30 13:04:00.526445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.526416    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.543722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.543689    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3a236d-openshift-default-dockercfg-lkhgs\""
Oct 30 13:04:00.550932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.550847    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"]
Oct 30 13:04:00.553943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.553903    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istiod-ca-cert\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.554117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.554086    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-token\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.554117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.554116    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-podinfo\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.554308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.554137    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-envoy\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.554308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.554178    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjcf\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-kube-api-access-7fjcf\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.554308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.554200    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-credential-socket\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.554308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.554279    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-socket\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.554308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.554295    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-certs\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.554577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.554386    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-data\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.658229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.657763    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-data\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.658229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.657884    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istiod-ca-cert\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.658229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.658090    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-token\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.658229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.658121    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-podinfo\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.658229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.658155    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-envoy\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.690318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.683301    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istiod-ca-cert\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.690318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.683597    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-data\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.703618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.703542    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-envoy\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.706304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.705268    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-podinfo\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.737225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.735270    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjcf\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-kube-api-access-7fjcf\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.737225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.735323    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-credential-socket\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.737225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.735426    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-socket\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.737225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.735451    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-certs\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.737225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.735885    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-certs\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.737225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.736066    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-credential-socket\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.737225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.736116    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-socket\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.745242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.744320    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-token\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.810297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.796346    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjcf\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-kube-api-access-7fjcf\") pod \"e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") " pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:00.857229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:00.854588    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:01.050328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.050141    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"]
Oct 30 13:04:01.054525 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.054476    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45be7029-76xx8"]
Oct 30 13:04:01.152059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.151973    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"
Oct 30 13:04:01.152059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.152010    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt" event={"ID":"82ba3b43-38f9-45c6-80c8-b4cb5e7f7209","Type":"ContainerDied","Data":"2833e8cff59d93a8fe97f2d5e93cd7c974fd58cc3744b22fe2fb1e8d3ac3b1d5"}
Oct 30 13:04:01.152357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.152088    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2833e8cff59d93a8fe97f2d5e93cd7c974fd58cc3744b22fe2fb1e8d3ac3b1d5"
Oct 30 13:04:01.192691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.192615    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"]
Oct 30 13:04:01.194685 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:04:01.194657    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ce1e86_cc1a_49bc_a3fe_cfe88b20fe9b.slice/crio-b39ca3872b04efbc8245213bb8ef7cf547bd4e0d83d2b50b3a02a74bd106374b WatchSource:0}: Error finding container b39ca3872b04efbc8245213bb8ef7cf547bd4e0d83d2b50b3a02a74bd106374b: Status 404 returned error can't find the container with id b39ca3872b04efbc8245213bb8ef7cf547bd4e0d83d2b50b3a02a74bd106374b
Oct 30 13:04:01.197658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.197545    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:01.197658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.197642    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:01.197821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.197687    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:01.711330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.711285    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358ac826-6788-4428-b685-9e485b695339" path="/var/lib/kubelet/pods/358ac826-6788-4428-b685-9e485b695339/volumes"
Oct 30 13:04:01.712066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:01.712007    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9c4d93-31ac-4675-9b26-8f342c320bf3" path="/var/lib/kubelet/pods/9b9c4d93-31ac-4675-9b26-8f342c320bf3/volumes"
Oct 30 13:04:02.166166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:02.166123    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" event={"ID":"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b","Type":"ContainerStarted","Data":"bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020"}
Oct 30 13:04:02.166385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:02.166197    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" event={"ID":"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b","Type":"ContainerStarted","Data":"b39ca3872b04efbc8245213bb8ef7cf547bd4e0d83d2b50b3a02a74bd106374b"}
Oct 30 13:04:02.261969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:02.261904    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" podStartSLOduration=2.261883856 podStartE2EDuration="2.261883856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:02.257381327 +0000 UTC m=+1906.733100936" watchObservedRunningTime="2026-10-30 13:04:02.261883856 +0000 UTC m=+1906.737603467"
Oct 30 13:04:02.857895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:02.857859    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:02.864340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:02.864307    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:03.223292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:03.219349    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:03.223292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:03.220470    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:08.575900 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:04:08.575853    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:04:15.038700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:15.038662    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"]
Oct 30 13:04:15.051308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:15.051275    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bbac5fe-vfr47"]
Oct 30 13:04:15.695038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:15.695001    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd12953c-eb02-4406-91d3-4fe27de8ecba" path="/var/lib/kubelet/pods/fd12953c-eb02-4406-91d3-4fe27de8ecba/volumes"
Oct 30 13:04:17.129433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:17.129395    4993 scope.go:122] "RemoveContainer" containerID="3a9f6e9a01c2b5bde871fc94c62ced0fa162a7d7e4589484a3632c00e5f872fc"
Oct 30 13:04:17.141912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:17.141876    4993 scope.go:122] "RemoveContainer" containerID="5591e5cfc20b3ec29831527367822afbd5099e11b7c8a2808b6be028c2ee74f5"
Oct 30 13:04:17.158943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:17.158915    4993 scope.go:122] "RemoveContainer" containerID="9c080ed93d17167206fcef3d5b800c8fd6067ed56182c69b993f8e2f0175967f"
Oct 30 13:04:17.170646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:17.170621    4993 scope.go:122] "RemoveContainer" containerID="83e075c1f38699ae5ba3d83accc4d8072e8cf3c84e2bd4f3fa9b3277355fee2c"
Oct 30 13:04:17.182425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:17.182397    4993 scope.go:122] "RemoveContainer" containerID="dcd5abe69e2c17d843ce89e57bbaac62d4436da096af70b546ed8ca75b47d1af"
Oct 30 13:04:17.197085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:17.196959    4993 scope.go:122] "RemoveContainer" containerID="9cff8a3aed5c0028a41e66b15b77c40d838cb5c7c5848d0f0dfe83bbfa50285f"
Oct 30 13:04:17.209386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:17.209344    4993 scope.go:122] "RemoveContainer" containerID="88e5daaf1b28fa9d4eb4b45d9beefcd5dd83460794f362bb9b0cd521f5937e34"
Oct 30 13:04:18.030906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:18.030862    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"]
Oct 30 13:04:18.046786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:18.046744    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2590b741-7jfcs"]
Oct 30 13:04:19.694144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:19.694102    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ee01c6-7655-49d9-bbf8-0b0762765221" path="/var/lib/kubelet/pods/32ee01c6-7655-49d9-bbf8-0b0762765221/volumes"
Oct 30 13:04:20.031726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:20.031688    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"]
Oct 30 13:04:20.043317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:20.043282    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a1c7b9-hvm9k"]
Oct 30 13:04:21.704235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:21.703379    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bfc4e95-38d8-41af-8e0f-637e9a068b20" path="/var/lib/kubelet/pods/2bfc4e95-38d8-41af-8e0f-637e9a068b20/volumes"
Oct 30 13:04:22.385176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:22.385131    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"]
Oct 30 13:04:22.385484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:22.385442    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy" containerID="cri-o://ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6" gracePeriod=30
Oct 30 13:04:27.694836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.694801    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 13:04:27.842180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842133    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965c37e2-a393-4b30-aa8d-35dc060595ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965c37e2-a393-4b30-aa8d-35dc060595ce-istiod-ca-cert\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.842421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842197    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-envoy\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.842421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842362    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-data\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.842512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842494    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-podinfo\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.842559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842519    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-certs\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.842559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842552    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-token\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-token\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.842634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842594    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-socket\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.842634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842605    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-data" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:27.842750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842730    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965c37e2-a393-4b30-aa8d-35dc060595ce-istiod-ca-cert" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:27.842750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842736    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-kube-api-access-8hb5j\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-kube-api-access-8hb5j\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.842871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842827    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-certs" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:27.842871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842836    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-socket" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:27.843310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.842838    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-credential-socket\") pod \"965c37e2-a393-4b30-aa8d-35dc060595ce\" (UID: \"965c37e2-a393-4b30-aa8d-35dc060595ce\") "
Oct 30 13:04:27.843310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.843125    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-credential-socket" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:27.843874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.843704    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965c37e2-a393-4b30-aa8d-35dc060595ce-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:27.843874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.843734    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:27.843874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.843748    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:27.843874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.843760    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:27.843874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.843772    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:27.845265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.845235    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-envoy" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:27.845361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.845264    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-podinfo" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:04:27.845403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.845352    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-kube-api-access-8hb5j" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "kube-api-access-8hb5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:27.845403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.845372    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-token" pod "965c37e2-a393-4b30-aa8d-35dc060595ce" (UID: "965c37e2-a393-4b30-aa8d-35dc060595ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:27.945799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.945698    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:27.945799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.945729    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:27.945799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.945738    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hb5j\" (UniqueName: \"kubernetes.io/projected/965c37e2-a393-4b30-aa8d-35dc060595ce-kube-api-access-8hb5j\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:27.945799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:27.945752    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965c37e2-a393-4b30-aa8d-35dc060595ce-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:28.037078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.037035    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"]
Oct 30 13:04:28.042103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.042065    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea617935-kcs4c"]
Oct 30 13:04:28.464959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.464917    4993 generic.go:356] "Generic (PLEG): container finished" podID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerID="ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6" exitCode=0
Oct 30 13:04:28.465190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.464969    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"
Oct 30 13:04:28.465190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.464969    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" event={"ID":"965c37e2-a393-4b30-aa8d-35dc060595ce","Type":"ContainerDied","Data":"ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6"}
Oct 30 13:04:28.465190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.465035    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr" event={"ID":"965c37e2-a393-4b30-aa8d-35dc060595ce","Type":"ContainerDied","Data":"9a31b4f4c96eae78af412450028605b871b35e5b1cbf14eff425e8ad3b94520a"}
Oct 30 13:04:28.465190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.465066    4993 scope.go:122] "RemoveContainer" containerID="ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6"
Oct 30 13:04:28.479670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.479631    4993 scope.go:122] "RemoveContainer" containerID="ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6"
Oct 30 13:04:28.479989 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:04:28.479966    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6\": container with ID starting with ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6 not found: ID does not exist" containerID="ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6"
Oct 30 13:04:28.480066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.479999    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6"} err="failed to get container status \"ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6\": rpc error: code = NotFound desc = could not find container \"ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6\": container with ID starting with ede04b732fc48e63ac4692632a4ff0dc793fc0c019232cb2b7ba701a3dd893b6 not found: ID does not exist"
Oct 30 13:04:28.520447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.520407    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"]
Oct 30 13:04:28.532927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:28.532885    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-cv4lr"]
Oct 30 13:04:29.694370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:29.694331    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" path="/var/lib/kubelet/pods/965c37e2-a393-4b30-aa8d-35dc060595ce/volumes"
Oct 30 13:04:29.694848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:29.694833    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9018a3-576b-4d77-859c-4c9fc233d394" path="/var/lib/kubelet/pods/cb9018a3-576b-4d77-859c-4c9fc233d394/volumes"
Oct 30 13:04:38.575385 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:04:38.575285    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:04:39.041281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:39.041232    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"]
Oct 30 13:04:39.056269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:39.056219    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eedec044-xrwsq"]
Oct 30 13:04:39.699018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:39.698981    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5628793-8ea3-4803-9c30-c25e98c635e6" path="/var/lib/kubelet/pods/d5628793-8ea3-4803-9c30-c25e98c635e6/volumes"
Oct 30 13:04:39.785180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:39.785140    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"]
Oct 30 13:04:39.796575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:39.796537    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a2768f-4xbpt"]
Oct 30 13:04:40.280259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:40.280173    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"]
Oct 30 13:04:40.280526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:40.280496    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy" containerID="cri-o://96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f" gracePeriod=30
Oct 30 13:04:41.609695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.609647    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"]
Oct 30 13:04:41.610574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.610550    4993 cpu_manager.go:414] "containerMap: removing container" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy"
Oct 30 13:04:41.610638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.610579    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy"
Oct 30 13:04:41.610826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.610813    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="965c37e2-a393-4b30-aa8d-35dc060595ce" containerName="istio-proxy"
Oct 30 13:04:41.614023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.613991    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.630537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.630510    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:41.642892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.642839    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:04:41.643085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.642927    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:04:41.643085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.643008    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:04:41.648196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.648162    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"]
Oct 30 13:04:41.696146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.696113    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ba3b43-38f9-45c6-80c8-b4cb5e7f7209" path="/var/lib/kubelet/pods/82ba3b43-38f9-45c6-80c8-b4cb5e7f7209/volumes"
Oct 30 13:04:41.702286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.702247    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztdh8\" (UniqueName: \"kubernetes.io/projected/a6d8667d-c6b0-4eb1-affe-2e7033a91391-kube-api-access-ztdh8\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.702464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.702321    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.702582 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.702556    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.803703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.803668    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdh8\" (UniqueName: \"kubernetes.io/projected/a6d8667d-c6b0-4eb1-affe-2e7033a91391-kube-api-access-ztdh8\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.803895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.803721    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.803895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.803822    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.804556 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.804526    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.806324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.806300    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.830729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.830682    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdh8\" (UniqueName: \"kubernetes.io/projected/a6d8667d-c6b0-4eb1-affe-2e7033a91391-kube-api-access-ztdh8\") pod \"maas-api-revoke-sub-b953eea9-cqh44\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:41.936293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:41.929487    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:42.382197 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:42.382155    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"]
Oct 30 13:04:42.382617 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:04:42.382578    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d8667d_c6b0_4eb1_affe_2e7033a91391.slice/crio-ae969eeefdd4cc2db0a2fb9eeb96a94ef4f75c3e784a30e8d1ca656a087a98d3 WatchSource:0}: Error finding container ae969eeefdd4cc2db0a2fb9eeb96a94ef4f75c3e784a30e8d1ca656a087a98d3: Status 404 returned error can't find the container with id ae969eeefdd4cc2db0a2fb9eeb96a94ef4f75c3e784a30e8d1ca656a087a98d3
Oct 30 13:04:42.648334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:42.648251    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44" event={"ID":"a6d8667d-c6b0-4eb1-affe-2e7033a91391","Type":"ContainerStarted","Data":"ae969eeefdd4cc2db0a2fb9eeb96a94ef4f75c3e784a30e8d1ca656a087a98d3"}
Oct 30 13:04:43.659993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:43.659892    4993 generic.go:356] "Generic (PLEG): container finished" podID="a6d8667d-c6b0-4eb1-affe-2e7033a91391" containerID="96f02303b28e2a212fc124bd61982671494e70224128307fc8d9a4d4d2e1bdaf" exitCode=0
Oct 30 13:04:43.659993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:43.659974    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44" event={"ID":"a6d8667d-c6b0-4eb1-affe-2e7033a91391","Type":"ContainerDied","Data":"96f02303b28e2a212fc124bd61982671494e70224128307fc8d9a4d4d2e1bdaf"}
Oct 30 13:04:43.662280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:43.662241    4993 generic.go:356] "Generic (PLEG): container finished" podID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerID="bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020" exitCode=0
Oct 30 13:04:43.662455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:43.662279    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" event={"ID":"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b","Type":"ContainerDied","Data":"bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020"}
Oct 30 13:04:43.662806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:43.662782    4993 scope.go:122] "RemoveContainer" containerID="bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020"
Oct 30 13:04:43.665593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:43.664550    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:43.665593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:43.664629    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:43.665593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:43.664678    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:44.681908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:44.681865    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" event={"ID":"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b","Type":"ContainerStarted","Data":"a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d"}
Oct 30 13:04:44.883510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:44.883460    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:44.884621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:44.884579    4993 patch_prober.go:29] interesting pod/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:04:44.884779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:44.884648    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 13:04:44.916907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:44.916878    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:45.086802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.086743    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-xks-ca\") pod \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") "
Oct 30 13:04:45.087014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.086837    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d8667d-c6b0-4eb1-affe-2e7033a91391-kube-api-access-ztdh8\" (UniqueName: \"kubernetes.io/projected/a6d8667d-c6b0-4eb1-affe-2e7033a91391-kube-api-access-ztdh8\") pod \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") "
Oct 30 13:04:45.087119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.087096    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-openshift-ca\") pod \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\" (UID: \"a6d8667d-c6b0-4eb1-affe-2e7033a91391\") "
Oct 30 13:04:45.087732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.087700    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-openshift-ca" pod "a6d8667d-c6b0-4eb1-affe-2e7033a91391" (UID: "a6d8667d-c6b0-4eb1-affe-2e7033a91391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:45.087893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.087837    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.089482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.089349    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-xks-ca" pod "a6d8667d-c6b0-4eb1-affe-2e7033a91391" (UID: "a6d8667d-c6b0-4eb1-affe-2e7033a91391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:45.090312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.089950    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d8667d-c6b0-4eb1-affe-2e7033a91391-kube-api-access-ztdh8" pod "a6d8667d-c6b0-4eb1-affe-2e7033a91391" (UID: "a6d8667d-c6b0-4eb1-affe-2e7033a91391"). InnerVolumeSpecName "kube-api-access-ztdh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:45.189242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.189188    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d8667d-c6b0-4eb1-affe-2e7033a91391-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.189242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.189238    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztdh8\" (UniqueName: \"kubernetes.io/projected/a6d8667d-c6b0-4eb1-affe-2e7033a91391-kube-api-access-ztdh8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.610943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.610915    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 13:04:45.706318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.702921    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d67554-66c0-4518-8818-6655f02c621f-istiod-ca-cert" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:45.707219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.702318    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d67554-66c0-4518-8818-6655f02c621f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d67554-66c0-4518-8818-6655f02c621f-istiod-ca-cert\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.707340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.707310    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-data\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.707410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.707376    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-istio-token\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-istio-token\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.707496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.707482    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-socket\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.707552 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.707542    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-credential-socket\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.707749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.707731    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-envoy\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.707837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.707822    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-certs\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.707906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.707861    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-kube-api-access-7w97d\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-kube-api-access-7w97d\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.708139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.708123    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45d67554-66c0-4518-8818-6655f02c621f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d67554-66c0-4518-8818-6655f02c621f-istio-podinfo\") pod \"45d67554-66c0-4518-8818-6655f02c621f\" (UID: \"45d67554-66c0-4518-8818-6655f02c621f\") "
Oct 30 13:04:45.709743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.709710    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d67554-66c0-4518-8818-6655f02c621f-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.711483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.711442    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-credential-socket" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.711693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.711662    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-data" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.711779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.711756    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-certs" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.711988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.711960    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-socket" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.716618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.716578    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-envoy" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.721799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.721751    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-istio-token" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:45.722153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.722125    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"
Oct 30 13:04:45.723700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.723668    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d67554-66c0-4518-8818-6655f02c621f-istio-podinfo" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:04:45.724145 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.724109    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44" event={"ID":"a6d8667d-c6b0-4eb1-affe-2e7033a91391","Type":"ContainerDied","Data":"ae969eeefdd4cc2db0a2fb9eeb96a94ef4f75c3e784a30e8d1ca656a087a98d3"}
Oct 30 13:04:45.724304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.724188    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae969eeefdd4cc2db0a2fb9eeb96a94ef4f75c3e784a30e8d1ca656a087a98d3"
Oct 30 13:04:45.727702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.727666    4993 generic.go:356] "Generic (PLEG): container finished" podID="45d67554-66c0-4518-8818-6655f02c621f" containerID="96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f" exitCode=0
Oct 30 13:04:45.728956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.728922    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" event={"ID":"45d67554-66c0-4518-8818-6655f02c621f","Type":"ContainerDied","Data":"96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f"}
Oct 30 13:04:45.729176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.728986    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9" event={"ID":"45d67554-66c0-4518-8818-6655f02c621f","Type":"ContainerDied","Data":"5dd04abe7286e5fdef8338c0a11740980db47554ba2910ed3861c5980d237992"}
Oct 30 13:04:45.729176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.729035    4993 scope.go:122] "RemoveContainer" containerID="96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f"
Oct 30 13:04:45.739373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.739195    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-kube-api-access-7w97d" pod "45d67554-66c0-4518-8818-6655f02c621f" (UID: "45d67554-66c0-4518-8818-6655f02c621f"). InnerVolumeSpecName "kube-api-access-7w97d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:45.739655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.739624    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"
Oct 30 13:04:45.785434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.785403    4993 scope.go:122] "RemoveContainer" containerID="96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f"
Oct 30 13:04:45.785924 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:04:45.785887    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f\": container with ID starting with 96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f not found: ID does not exist" containerID="96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f"
Oct 30 13:04:45.786029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.785938    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f"} err="failed to get container status \"96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f\": rpc error: code = NotFound desc = could not find container \"96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f\": container with ID starting with 96617b5f704dd1b744cd4c03cca6516845e0efeacbff3c88e7b9ae37513c5b4f not found: ID does not exist"
Oct 30 13:04:45.808937 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:04:45.808894    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d67554_66c0_4518_8818_6655f02c621f.slice/crio-5dd04abe7286e5fdef8338c0a11740980db47554ba2910ed3861c5980d237992\": RecentStats: unable to find data in memory cache]"
Oct 30 13:04:45.811149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.811003    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.811149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.811044    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w97d\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-kube-api-access-7w97d\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.811149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.811059    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d67554-66c0-4518-8818-6655f02c621f-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.811149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.811072    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.811149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.811084    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d67554-66c0-4518-8818-6655f02c621f-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.811149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.811097    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.811149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.811110    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.811149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.811122    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d67554-66c0-4518-8818-6655f02c621f-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.869245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.863633    4993 patch_prober.go:29] interesting pod/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:04:45.886633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.869514    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 13:04:45.892322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.892274    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"]
Oct 30 13:04:45.896006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:45.895965    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-slpm9"]
Oct 30 13:04:46.862637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:46.862598    4993 patch_prober.go:29] interesting pod/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:04:46.913818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:46.913777    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 13:04:47.697614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:47.697575    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d67554-66c0-4518-8818-6655f02c621f" path="/var/lib/kubelet/pods/45d67554-66c0-4518-8818-6655f02c621f/volumes"
Oct 30 13:04:47.858708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:47.858656    4993 patch_prober.go:29] interesting pod/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:04:47.858936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:47.858745    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 13:04:48.226377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:48.226326    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:48.869233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:48.863631    4993 patch_prober.go:29] interesting pod/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:04:48.869233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:48.863699    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 13:04:49.873233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:49.871751    4993 patch_prober.go:29] interesting pod/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:04:49.873233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:49.871816    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 13:04:50.329231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.317577    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"]
Oct 30 13:04:50.335243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.332745    4993 cpu_manager.go:414] "containerMap: removing container" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy"
Oct 30 13:04:50.335243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.332774    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy"
Oct 30 13:04:50.335243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.332851    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a6d8667d-c6b0-4eb1-affe-2e7033a91391" containerName="revoke-keys"
Oct 30 13:04:50.335243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.332859    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d8667d-c6b0-4eb1-affe-2e7033a91391" containerName="revoke-keys"
Oct 30 13:04:50.335243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.333003    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d8667d-c6b0-4eb1-affe-2e7033a91391" containerName="revoke-keys"
Oct 30 13:04:50.335243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.333016    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d67554-66c0-4518-8818-6655f02c621f" containerName="istio-proxy"
Oct 30 13:04:50.353584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.353539    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.389955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.375454    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"]
Oct 30 13:04:50.389955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.381593    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fa855c-openshift-default-dockercfg-5f4qm\""
Oct 30 13:04:50.437552 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437505    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-data\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.437772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437564    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-podinfo\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.437772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437632    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a0d1be-3eed-44bd-8358-e091469a84ea-istiod-ca-cert\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.437772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437714    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-socket\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.437772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437739    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-certs\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.437772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437762    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-token\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.438041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437791    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-envoy\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.438041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437832    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-credential-socket\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.438041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.437858    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx7f9\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-kube-api-access-rx7f9\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539433    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-credential-socket\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539495    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx7f9\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-kube-api-access-rx7f9\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539672    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-data\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539828 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539706    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-podinfo\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539828 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539768    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a0d1be-3eed-44bd-8358-e091469a84ea-istiod-ca-cert\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539846    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-socket\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539871    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-certs\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539897    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-token\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.539944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.539926    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-envoy\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.544221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.544154    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-credential-socket\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.545032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.545006    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-data\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.591282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.567522    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a0d1be-3eed-44bd-8358-e091469a84ea-istiod-ca-cert\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.591282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.567902    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-socket\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.591282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.573587    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-certs\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.591282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.574160    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-envoy\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.591282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.580459    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx7f9\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-kube-api-access-rx7f9\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.592587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.592385    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-podinfo\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.626218 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.626169    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-token\") pod \"e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") " pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.716950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.716912    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:50.863330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.863187    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:50.864584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:50.864559    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:04:51.076032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:51.075806    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"]
Oct 30 13:04:51.076602 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:04:51.076561    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a0d1be_3eed_44bd_8358_e091469a84ea.slice/crio-c6fc33c156da7d70ccbed7f09d1ae1990a63a8c50c2bbe82bb44b7430f4723fd WatchSource:0}: Error finding container c6fc33c156da7d70ccbed7f09d1ae1990a63a8c50c2bbe82bb44b7430f4723fd: Status 404 returned error can't find the container with id c6fc33c156da7d70ccbed7f09d1ae1990a63a8c50c2bbe82bb44b7430f4723fd
Oct 30 13:04:51.079571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:51.079531    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:51.079713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:51.079623    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:51.079713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:51.079670    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:51.822455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:51.822403    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" event={"ID":"b9a0d1be-3eed-44bd-8358-e091469a84ea","Type":"ContainerStarted","Data":"749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076"}
Oct 30 13:04:51.822670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:51.822477    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" event={"ID":"b9a0d1be-3eed-44bd-8358-e091469a84ea","Type":"ContainerStarted","Data":"c6fc33c156da7d70ccbed7f09d1ae1990a63a8c50c2bbe82bb44b7430f4723fd"}
Oct 30 13:04:51.873325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:51.873271    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podStartSLOduration=1.8732519490000001 podStartE2EDuration="1.873251949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:51.87094011 +0000 UTC m=+1956.346659720" watchObservedRunningTime="2026-10-30 13:04:51.873251949 +0000 UTC m=+1956.348971561"
Oct 30 13:04:52.717082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:52.717049    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:52.722955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:52.722923    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:52.831695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:52.831661    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:52.832724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:52.832694    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:04:53.953412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:53.953373    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"]
Oct 30 13:04:53.957594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:53.957566    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:53.987443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:53.987401    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f9c553-openshift-default-dockercfg-5zvwc\""
Oct 30 13:04:54.007385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.007337    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.007672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.007637    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"]
Oct 30 13:04:54.007808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.007644    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istiod-ca-cert\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.007808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.007751    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-certs\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.007808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.007803    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-data\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.007958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.007822    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-envoy\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.007958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.007941    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-podinfo\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.008068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.008050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgr7f\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-kube-api-access-tgr7f\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.008527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.008123    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-credential-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.008527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.008148    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-token\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.040557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.040510    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"]
Oct 30 13:04:54.044057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.044026    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a6d79d-c7h5b"]
Oct 30 13:04:54.119230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.118610    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-credential-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.119230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.118678    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-token\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.119230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.118812    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.119230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.118852    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istiod-ca-cert\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.119230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.118897    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-certs\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.119230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.118981    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-data\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.119230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.119009    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-envoy\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.119230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.119084    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-podinfo\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.119828 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.119249    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgr7f\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-kube-api-access-tgr7f\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.124239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.120110    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-credential-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.124239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.120644    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.133636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.133597    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-data\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.133886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.133863    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-certs\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.134368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.134329    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istiod-ca-cert\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.155231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.153142    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgr7f\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-kube-api-access-tgr7f\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.163229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.160837    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-token\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.188234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.174719    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-envoy\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.188234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.174941    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-podinfo\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.306233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.305639    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:54.702332 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:04:54.702295    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbe7dff_ab80_448b_b0d2_076e434a5cfd.slice/crio-c859d57ae0f3728e7a1a38f656f5b98d4e86d7628897c19d428b472ac1ce1482 WatchSource:0}: Error finding container c859d57ae0f3728e7a1a38f656f5b98d4e86d7628897c19d428b472ac1ce1482: Status 404 returned error can't find the container with id c859d57ae0f3728e7a1a38f656f5b98d4e86d7628897c19d428b472ac1ce1482
Oct 30 13:04:54.703035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.703008    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"]
Oct 30 13:04:54.705389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.705351    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:54.705512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.705438    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:54.705512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.705485    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:54.861536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.861494    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" event={"ID":"0cbe7dff-ab80-448b-b0d2-076e434a5cfd","Type":"ContainerStarted","Data":"3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8"}
Oct 30 13:04:54.861721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.861559    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" event={"ID":"0cbe7dff-ab80-448b-b0d2-076e434a5cfd","Type":"ContainerStarted","Data":"c859d57ae0f3728e7a1a38f656f5b98d4e86d7628897c19d428b472ac1ce1482"}
Oct 30 13:04:54.913998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:54.913931    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" podStartSLOduration=1.9139153869999999 podStartE2EDuration="1.913915387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:54.909444628 +0000 UTC m=+1959.385164238" watchObservedRunningTime="2026-10-30 13:04:54.913915387 +0000 UTC m=+1959.389635043"
Oct 30 13:04:55.306663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:55.306629    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:55.313031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:55.313000    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:55.694419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:55.694328    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8888693a-b071-45e0-af82-421962cd2416" path="/var/lib/kubelet/pods/8888693a-b071-45e0-af82-421962cd2416/volumes"
Oct 30 13:04:55.868454 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:55.868414    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:04:55.869745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:04:55.869719    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:05:08.571517 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:05:08.571473    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:05:17.299454 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.299413    4993 scope.go:122] "RemoveContainer" containerID="8ba77f6a8f7a94d77faff3362204d3a99085ebaa9945169d3643578745fe1e17"
Oct 30 13:05:17.313611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.313582    4993 scope.go:122] "RemoveContainer" containerID="1369a5fe5283f951811e352d3acc818857fce8e33e1eab81d52f4faae3ba3020"
Oct 30 13:05:17.324758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.324732    4993 scope.go:122] "RemoveContainer" containerID="50b9619144fddc0ebdefe6bab3fa4686b5b03a7a94b41e7f92b1162dbe3e647e"
Oct 30 13:05:17.336017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.335990    4993 scope.go:122] "RemoveContainer" containerID="22ab588898ae9db944c436ce08e4cb1a5d42a364c7f505a9d979fd0fdd8af38d"
Oct 30 13:05:17.348593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.348569    4993 scope.go:122] "RemoveContainer" containerID="30ed7a5cd5b3d0092b9011f9db57c5d3b986f63699f0cd312468f4e96c47a255"
Oct 30 13:05:17.359638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.359608    4993 scope.go:122] "RemoveContainer" containerID="00c9740e1f52ef973abea646217caf65b9960b7459f90b64239a30913219e076"
Oct 30 13:05:17.372738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.372710    4993 scope.go:122] "RemoveContainer" containerID="ac36c9657e0940fd8a5ba68b452a13706931361c3f31ffaae690bfbcc085d8a1"
Oct 30 13:05:17.384136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.384109    4993 scope.go:122] "RemoveContainer" containerID="6298ba842145b6af8690ddcd3598ea7c4f63a9396ccb64993fc8fb426bdbc506"
Oct 30 13:05:17.397615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.397590    4993 scope.go:122] "RemoveContainer" containerID="ee104fd9fc34e49595a0cca388fe1531328c688028224b3e01cf7d35949ac9d8"
Oct 30 13:05:17.408624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.408601    4993 scope.go:122] "RemoveContainer" containerID="5430af644e85fb1a687404219d83cfe21e994993a777714f3b4fc3ad45a44422"
Oct 30 13:05:17.419649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.419623    4993 scope.go:122] "RemoveContainer" containerID="40085b077bcb55a99768fdbffb3461e0b40a2423347d53a470893a5a85b90fd8"
Oct 30 13:05:17.430303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:17.430275    4993 scope.go:122] "RemoveContainer" containerID="579e8e1e8d263bd37c4cc7818fdafafd8252c4371dd44ac90a8d4baa4f0c9885"
Oct 30 13:05:22.282178 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.282134    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"]
Oct 30 13:05:22.288120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.288091    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.308973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.308926    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"]
Oct 30 13:05:22.318401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.318339    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:22.323545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.323507    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:05:22.323751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.323573    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:05:22.323942 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.323918    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:05:22.372612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.372570    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.372803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.372780    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.372894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.372825    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vchs7\" (UniqueName: \"kubernetes.io/projected/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-kube-api-access-vchs7\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.473658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.473613    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.473874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.473735    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.473874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.473768    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vchs7\" (UniqueName: \"kubernetes.io/projected/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-kube-api-access-vchs7\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.474450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.474413    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.476343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.476319    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.497655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.497617    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vchs7\" (UniqueName: \"kubernetes.io/projected/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-kube-api-access-vchs7\") pod \"maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.610348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.610247    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:22.925118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:22.925085    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"]
Oct 30 13:05:22.925844 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:05:22.925815    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96f847b_2c02_4cb7_9b91_dbcb0b3b41da.slice/crio-de39de7760be20f8268d23f9b395638edce348d80f44b2214555652be6b3781c WatchSource:0}: Error finding container de39de7760be20f8268d23f9b395638edce348d80f44b2214555652be6b3781c: Status 404 returned error can't find the container with id de39de7760be20f8268d23f9b395638edce348d80f44b2214555652be6b3781c
Oct 30 13:05:23.409396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:23.409351    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx" event={"ID":"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da","Type":"ContainerStarted","Data":"2530da77b85879bbbab288013e5d03291a6f9a7f0f3f62d1db4131a52f04c2bb"}
Oct 30 13:05:23.409868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:23.409414    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx" event={"ID":"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da","Type":"ContainerStarted","Data":"de39de7760be20f8268d23f9b395638edce348d80f44b2214555652be6b3781c"}
Oct 30 13:05:23.461779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:23.461668    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx" podStartSLOduration=1.461651768 podStartE2EDuration="1.461651768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:23.460435034 +0000 UTC m=+1987.936154659" watchObservedRunningTime="2026-10-30 13:05:23.461651768 +0000 UTC m=+1987.937371377"
Oct 30 13:05:24.432231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:24.431991    4993 generic.go:356] "Generic (PLEG): container finished" podID="c96f847b-2c02-4cb7-9b91-dbcb0b3b41da" containerID="2530da77b85879bbbab288013e5d03291a6f9a7f0f3f62d1db4131a52f04c2bb" exitCode=0
Oct 30 13:05:24.432231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:24.432124    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx" event={"ID":"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da","Type":"ContainerDied","Data":"2530da77b85879bbbab288013e5d03291a6f9a7f0f3f62d1db4131a52f04c2bb"}
Oct 30 13:05:25.612338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.612306    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:25.635871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.635832    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-xks-ca\") pod \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") "
Oct 30 13:05:25.636091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.635997    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-openshift-ca\") pod \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") "
Oct 30 13:05:25.636174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.636088    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-kube-api-access-vchs7\" (UniqueName: \"kubernetes.io/projected/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-kube-api-access-vchs7\") pod \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\" (UID: \"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da\") "
Oct 30 13:05:25.636393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.636364    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-openshift-ca" pod "c96f847b-2c02-4cb7-9b91-dbcb0b3b41da" (UID: "c96f847b-2c02-4cb7-9b91-dbcb0b3b41da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:25.636845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.636818    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:25.638646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.638615    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-xks-ca" pod "c96f847b-2c02-4cb7-9b91-dbcb0b3b41da" (UID: "c96f847b-2c02-4cb7-9b91-dbcb0b3b41da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:25.638745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.638620    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-kube-api-access-vchs7" pod "c96f847b-2c02-4cb7-9b91-dbcb0b3b41da" (UID: "c96f847b-2c02-4cb7-9b91-dbcb0b3b41da"). InnerVolumeSpecName "kube-api-access-vchs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:25.738221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.738181    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vchs7\" (UniqueName: \"kubernetes.io/projected/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-kube-api-access-vchs7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:25.738221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:25.738228    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:26.450032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:26.450002    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"
Oct 30 13:05:26.450237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:26.450037    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx" event={"ID":"c96f847b-2c02-4cb7-9b91-dbcb0b3b41da","Type":"ContainerDied","Data":"de39de7760be20f8268d23f9b395638edce348d80f44b2214555652be6b3781c"}
Oct 30 13:05:26.450237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:26.450084    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de39de7760be20f8268d23f9b395638edce348d80f44b2214555652be6b3781c"
Oct 30 13:05:35.083886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.083837    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"]
Oct 30 13:05:35.084689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.084666    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c96f847b-2c02-4cb7-9b91-dbcb0b3b41da" containerName="revoke-keys"
Oct 30 13:05:35.084689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.084690    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96f847b-2c02-4cb7-9b91-dbcb0b3b41da" containerName="revoke-keys"
Oct 30 13:05:35.084813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.084803    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96f847b-2c02-4cb7-9b91-dbcb0b3b41da" containerName="revoke-keys"
Oct 30 13:05:35.092199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.092151    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.109766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.109606    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"]
Oct 30 13:05:35.111340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.111314    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:05:35.111544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.111523    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:35.121998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.121967    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:05:35.122152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.122015    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:05:35.239524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.239476    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lz6\" (UniqueName: \"kubernetes.io/projected/95c30b99-a08e-44b3-b0a8-d045d217683a-kube-api-access-k7lz6\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.239695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.239666    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.239767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.239702    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.341317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.341199    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.341317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.341256    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.341575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.341377    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lz6\" (UniqueName: \"kubernetes.io/projected/95c30b99-a08e-44b3-b0a8-d045d217683a-kube-api-access-k7lz6\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.341897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.341875    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.343847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.343815    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.376232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.375328    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lz6\" (UniqueName: \"kubernetes.io/projected/95c30b99-a08e-44b3-b0a8-d045d217683a-kube-api-access-k7lz6\") pod \"maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.422349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.422312    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:35.795720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:35.795674    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"]
Oct 30 13:05:36.543858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:36.543813    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x" event={"ID":"95c30b99-a08e-44b3-b0a8-d045d217683a","Type":"ContainerStarted","Data":"6bfc4d6118ce55fd5bb08fab87c577a63b8bbbb7c62a0e363e4d784872e0baab"}
Oct 30 13:05:36.544321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:36.543883    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x" event={"ID":"95c30b99-a08e-44b3-b0a8-d045d217683a","Type":"ContainerStarted","Data":"1255b82b762f02b8150edcf27a86b4d64dc980d59af101bdfee1f349bb07400d"}
Oct 30 13:05:36.588469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:36.588393    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x" podStartSLOduration=1.58837048 podStartE2EDuration="1.58837048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:36.58506407 +0000 UTC m=+2001.060783680" watchObservedRunningTime="2026-10-30 13:05:36.58837048 +0000 UTC m=+2001.064090090"
Oct 30 13:05:37.553840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:37.553781    4993 generic.go:356] "Generic (PLEG): container finished" podID="95c30b99-a08e-44b3-b0a8-d045d217683a" containerID="6bfc4d6118ce55fd5bb08fab87c577a63b8bbbb7c62a0e363e4d784872e0baab" exitCode=0
Oct 30 13:05:37.554275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:37.553864    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x" event={"ID":"95c30b99-a08e-44b3-b0a8-d045d217683a","Type":"ContainerDied","Data":"6bfc4d6118ce55fd5bb08fab87c577a63b8bbbb7c62a0e363e4d784872e0baab"}
Oct 30 13:05:38.583889 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:05:38.583820    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:05:38.769344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.769312    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:38.874339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.874245    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:38.874931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.874622    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:38.923863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.923811    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-openshift-ca\") pod \"95c30b99-a08e-44b3-b0a8-d045d217683a\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") "
Oct 30 13:05:38.924050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.924023    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c30b99-a08e-44b3-b0a8-d045d217683a-kube-api-access-k7lz6\" (UniqueName: \"kubernetes.io/projected/95c30b99-a08e-44b3-b0a8-d045d217683a-kube-api-access-k7lz6\") pod \"95c30b99-a08e-44b3-b0a8-d045d217683a\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") "
Oct 30 13:05:38.924148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.924130    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-xks-ca\") pod \"95c30b99-a08e-44b3-b0a8-d045d217683a\" (UID: \"95c30b99-a08e-44b3-b0a8-d045d217683a\") "
Oct 30 13:05:38.924232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.924183    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-openshift-ca" pod "95c30b99-a08e-44b3-b0a8-d045d217683a" (UID: "95c30b99-a08e-44b3-b0a8-d045d217683a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:38.924720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.924695    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:38.926421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.926391    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-xks-ca" pod "95c30b99-a08e-44b3-b0a8-d045d217683a" (UID: "95c30b99-a08e-44b3-b0a8-d045d217683a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:38.926629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:38.926605    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c30b99-a08e-44b3-b0a8-d045d217683a-kube-api-access-k7lz6" pod "95c30b99-a08e-44b3-b0a8-d045d217683a" (UID: "95c30b99-a08e-44b3-b0a8-d045d217683a"). InnerVolumeSpecName "kube-api-access-k7lz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:39.025917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:39.025869    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7lz6\" (UniqueName: \"kubernetes.io/projected/95c30b99-a08e-44b3-b0a8-d045d217683a-kube-api-access-k7lz6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:39.025917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:39.025915    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c30b99-a08e-44b3-b0a8-d045d217683a-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:39.625127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:39.625069    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x" event={"ID":"95c30b99-a08e-44b3-b0a8-d045d217683a","Type":"ContainerDied","Data":"1255b82b762f02b8150edcf27a86b4d64dc980d59af101bdfee1f349bb07400d"}
Oct 30 13:05:39.625909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:39.625865    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1255b82b762f02b8150edcf27a86b4d64dc980d59af101bdfee1f349bb07400d"
Oct 30 13:05:39.626243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:39.626223    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"
Oct 30 13:05:39.873492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:39.873351    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"]
Oct 30 13:05:39.879869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:39.879757    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9368ce-tndzx"]
Oct 30 13:05:41.651666 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:41.651556    4993 generic.go:356] "Generic (PLEG): container finished" podID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerID="749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076" exitCode=0
Oct 30 13:05:41.652194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:41.651880    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" event={"ID":"b9a0d1be-3eed-44bd-8358-e091469a84ea","Type":"ContainerDied","Data":"749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076"}
Oct 30 13:05:41.652625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:41.652604    4993 scope.go:122] "RemoveContainer" containerID="749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076"
Oct 30 13:05:41.654810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:41.654770    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:41.654892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:41.654874    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:41.654945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:41.654930    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:41.747245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:41.747091    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96f847b-2c02-4cb7-9b91-dbcb0b3b41da" path="/var/lib/kubelet/pods/c96f847b-2c02-4cb7-9b91-dbcb0b3b41da/volumes"
Oct 30 13:05:42.681232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:42.676091    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" event={"ID":"b9a0d1be-3eed-44bd-8358-e091469a84ea","Type":"ContainerStarted","Data":"c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba"}
Oct 30 13:05:42.718033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:42.717998    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:05:42.718314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:42.718289    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:05:42.718369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:42.718339    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 13:05:43.717693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:43.717646    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:05:43.718140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:43.717722    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 13:05:44.718191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:44.718141    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:05:44.718668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:44.718238    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 13:05:45.717301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:45.717260    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:05:45.717511 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:45.717322    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 13:05:46.717288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:46.717256    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:05:46.717751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:46.717727    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 13:05:47.718252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:47.718189    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:05:47.718782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:47.718292    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 13:05:48.033267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.033192    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"]
Oct 30 13:05:48.033834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.033818    4993 cpu_manager.go:414] "containerMap: removing container" podUID="95c30b99-a08e-44b3-b0a8-d045d217683a" containerName="revoke-keys"
Oct 30 13:05:48.033885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.033835    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c30b99-a08e-44b3-b0a8-d045d217683a" containerName="revoke-keys"
Oct 30 13:05:48.033956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.033945    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c30b99-a08e-44b3-b0a8-d045d217683a" containerName="revoke-keys"
Oct 30 13:05:48.037036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.037011    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.066252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.065507    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:05:48.066252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.065611    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:05:48.074541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.074508    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"]
Oct 30 13:05:48.096739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.096706    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:05:48.097001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.096985    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:48.132283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.132245    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sghv\" (UniqueName: \"kubernetes.io/projected/dca9db10-2972-4c80-bf24-037b01f7f2fd-kube-api-access-4sghv\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.132467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.132353    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.132535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.132510    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.234316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.234257    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.234555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.234468    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.234629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.234590    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sghv\" (UniqueName: \"kubernetes.io/projected/dca9db10-2972-4c80-bf24-037b01f7f2fd-kube-api-access-4sghv\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.235170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.235133    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.237796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.237762    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.260660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.260625    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sghv\" (UniqueName: \"kubernetes.io/projected/dca9db10-2972-4c80-bf24-037b01f7f2fd-kube-api-access-4sghv\") pod \"maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.352356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.352184    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:48.722398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.722302    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:05:48.722894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.722872    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:05:48.723799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.723776    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:05:48.902228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:48.897329    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"]
Oct 30 13:05:49.751616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:49.751580    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9" event={"ID":"dca9db10-2972-4c80-bf24-037b01f7f2fd","Type":"ContainerStarted","Data":"817bd2206440cb0816573667bb6111eb173436dc90630217daade169002cac80"}
Oct 30 13:05:49.752133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:49.751637    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9" event={"ID":"dca9db10-2972-4c80-bf24-037b01f7f2fd","Type":"ContainerStarted","Data":"9aed427852a697660f7a56b6f11814d8be2c0f5451b5e54adffa3e51cb381215"}
Oct 30 13:05:49.813522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:49.813463    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9" podStartSLOduration=1.8134463790000002 podStartE2EDuration="1.813446379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:49.810009389 +0000 UTC m=+2014.285728998" watchObservedRunningTime="2026-10-30 13:05:49.813446379 +0000 UTC m=+2014.289165990"
Oct 30 13:05:50.771497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:50.771459    4993 generic.go:356] "Generic (PLEG): container finished" podID="dca9db10-2972-4c80-bf24-037b01f7f2fd" containerID="817bd2206440cb0816573667bb6111eb173436dc90630217daade169002cac80" exitCode=0
Oct 30 13:05:50.771950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:50.771554    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9" event={"ID":"dca9db10-2972-4c80-bf24-037b01f7f2fd","Type":"ContainerDied","Data":"817bd2206440cb0816573667bb6111eb173436dc90630217daade169002cac80"}
Oct 30 13:05:52.021400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.021370    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:52.130080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.129965    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca9db10-2972-4c80-bf24-037b01f7f2fd-kube-api-access-4sghv\" (UniqueName: \"kubernetes.io/projected/dca9db10-2972-4c80-bf24-037b01f7f2fd-kube-api-access-4sghv\") pod \"dca9db10-2972-4c80-bf24-037b01f7f2fd\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") "
Oct 30 13:05:52.130080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.130076    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-xks-ca\") pod \"dca9db10-2972-4c80-bf24-037b01f7f2fd\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") "
Oct 30 13:05:52.130369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.130121    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-openshift-ca\") pod \"dca9db10-2972-4c80-bf24-037b01f7f2fd\" (UID: \"dca9db10-2972-4c80-bf24-037b01f7f2fd\") "
Oct 30 13:05:52.131043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.131010    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-openshift-ca" pod "dca9db10-2972-4c80-bf24-037b01f7f2fd" (UID: "dca9db10-2972-4c80-bf24-037b01f7f2fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:52.132979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.132939    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-xks-ca" pod "dca9db10-2972-4c80-bf24-037b01f7f2fd" (UID: "dca9db10-2972-4c80-bf24-037b01f7f2fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:52.133114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.133025    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca9db10-2972-4c80-bf24-037b01f7f2fd-kube-api-access-4sghv" pod "dca9db10-2972-4c80-bf24-037b01f7f2fd" (UID: "dca9db10-2972-4c80-bf24-037b01f7f2fd"). InnerVolumeSpecName "kube-api-access-4sghv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:52.232793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.232755    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sghv\" (UniqueName: \"kubernetes.io/projected/dca9db10-2972-4c80-bf24-037b01f7f2fd-kube-api-access-4sghv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:52.232793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.232792    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:52.233020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.232806    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dca9db10-2972-4c80-bf24-037b01f7f2fd-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:52.822270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.822202    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"
Oct 30 13:05:52.822492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.822263    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9" event={"ID":"dca9db10-2972-4c80-bf24-037b01f7f2fd","Type":"ContainerDied","Data":"9aed427852a697660f7a56b6f11814d8be2c0f5451b5e54adffa3e51cb381215"}
Oct 30 13:05:52.822492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:52.822322    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aed427852a697660f7a56b6f11814d8be2c0f5451b5e54adffa3e51cb381215"
Oct 30 13:05:54.434049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.430963    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"]
Oct 30 13:05:54.434049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.431878    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dca9db10-2972-4c80-bf24-037b01f7f2fd" containerName="revoke-keys"
Oct 30 13:05:54.434049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.431901    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca9db10-2972-4c80-bf24-037b01f7f2fd" containerName="revoke-keys"
Oct 30 13:05:54.434049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.432076    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca9db10-2972-4c80-bf24-037b01f7f2fd" containerName="revoke-keys"
Oct 30 13:05:54.436906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.435831    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.484916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.484865    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"]
Oct 30 13:05:54.495537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.495497    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:05:54.495752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.495579    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:05:54.539263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.537714    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:05:54.539263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.537805    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:54.579952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.579900    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.579952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.579956    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.580248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.580198    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nnls\" (UniqueName: \"kubernetes.io/projected/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-kube-api-access-7nnls\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.681680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.681617    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nnls\" (UniqueName: \"kubernetes.io/projected/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-kube-api-access-7nnls\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.681884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.681792    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.681936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.681903    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.682694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.682660    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.684692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.684631    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.706167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.706133    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nnls\" (UniqueName: \"kubernetes.io/projected/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-kube-api-access-7nnls\") pod \"maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:54.782633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:54.782588    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:55.116146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:55.116110    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"]
Oct 30 13:05:55.117061 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:05:55.117029    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce974da_ff6a_4e52_97ed_ef34f5a7ff36.slice/crio-9f8cb2d41435152a17c7eb2ef5205abaf0f8cd5242077e53eca67b65043859ec WatchSource:0}: Error finding container 9f8cb2d41435152a17c7eb2ef5205abaf0f8cd5242077e53eca67b65043859ec: Status 404 returned error can't find the container with id 9f8cb2d41435152a17c7eb2ef5205abaf0f8cd5242077e53eca67b65043859ec
Oct 30 13:05:55.859036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:55.858998    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr" event={"ID":"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36","Type":"ContainerStarted","Data":"03e8ff84aaf2a8ecdf5ece47b0fc0f345f37c9578150ed30a4044bd609185e04"}
Oct 30 13:05:55.859578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:55.859060    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr" event={"ID":"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36","Type":"ContainerStarted","Data":"9f8cb2d41435152a17c7eb2ef5205abaf0f8cd5242077e53eca67b65043859ec"}
Oct 30 13:05:55.906159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:55.906093    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr" podStartSLOduration=1.9060748379999999 podStartE2EDuration="1.906074838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:55.903775169 +0000 UTC m=+2020.379494807" watchObservedRunningTime="2026-10-30 13:05:55.906074838 +0000 UTC m=+2020.381794491"
Oct 30 13:05:56.868831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:56.868789    4993 generic.go:356] "Generic (PLEG): container finished" podID="7ce974da-ff6a-4e52-97ed-ef34f5a7ff36" containerID="03e8ff84aaf2a8ecdf5ece47b0fc0f345f37c9578150ed30a4044bd609185e04" exitCode=0
Oct 30 13:05:56.869321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:56.869070    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr" event={"ID":"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36","Type":"ContainerDied","Data":"03e8ff84aaf2a8ecdf5ece47b0fc0f345f37c9578150ed30a4044bd609185e04"}
Oct 30 13:05:58.084068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.084036    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:58.237477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.237442    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-kube-api-access-7nnls\" (UniqueName: \"kubernetes.io/projected/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-kube-api-access-7nnls\") pod \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") "
Oct 30 13:05:58.237679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.237556    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-openshift-ca\") pod \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") "
Oct 30 13:05:58.237679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.237643    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-xks-ca\") pod \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\" (UID: \"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36\") "
Oct 30 13:05:58.237939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.237912    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-openshift-ca" pod "7ce974da-ff6a-4e52-97ed-ef34f5a7ff36" (UID: "7ce974da-ff6a-4e52-97ed-ef34f5a7ff36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:58.238286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.238267    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:58.239883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.239851    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-xks-ca" pod "7ce974da-ff6a-4e52-97ed-ef34f5a7ff36" (UID: "7ce974da-ff6a-4e52-97ed-ef34f5a7ff36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:58.239974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.239896    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-kube-api-access-7nnls" pod "7ce974da-ff6a-4e52-97ed-ef34f5a7ff36" (UID: "7ce974da-ff6a-4e52-97ed-ef34f5a7ff36"). InnerVolumeSpecName "kube-api-access-7nnls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:58.339325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.339282    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nnls\" (UniqueName: \"kubernetes.io/projected/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-kube-api-access-7nnls\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:58.339325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:58.339318    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:05:59.127954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:59.112451    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr" event={"ID":"7ce974da-ff6a-4e52-97ed-ef34f5a7ff36","Type":"ContainerDied","Data":"9f8cb2d41435152a17c7eb2ef5205abaf0f8cd5242077e53eca67b65043859ec"}
Oct 30 13:05:59.127954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:59.112562    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f8cb2d41435152a17c7eb2ef5205abaf0f8cd5242077e53eca67b65043859ec"
Oct 30 13:05:59.127954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:59.112648    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"
Oct 30 13:05:59.139132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:59.139090    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"]
Oct 30 13:05:59.146187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:59.146133    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80970991-8js42"]
Oct 30 13:05:59.701182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:05:59.701137    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2565a5c8-ca9c-4600-8cab-8b84aba412c5" path="/var/lib/kubelet/pods/2565a5c8-ca9c-4600-8cab-8b84aba412c5/volumes"
Oct 30 13:06:01.119868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:01.119828    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"]
Oct 30 13:06:01.132674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:01.132635    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6f0f50-tx88x"]
Oct 30 13:06:01.692501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:01.692465    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c30b99-a08e-44b3-b0a8-d045d217683a" path="/var/lib/kubelet/pods/95c30b99-a08e-44b3-b0a8-d045d217683a/volumes"
Oct 30 13:06:08.580988 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:06:08.580938    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:06:09.502342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.502299    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"]
Oct 30 13:06:09.502999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.502975    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7ce974da-ff6a-4e52-97ed-ef34f5a7ff36" containerName="revoke-keys"
Oct 30 13:06:09.503129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.503001    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce974da-ff6a-4e52-97ed-ef34f5a7ff36" containerName="revoke-keys"
Oct 30 13:06:09.503241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.503227    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce974da-ff6a-4e52-97ed-ef34f5a7ff36" containerName="revoke-keys"
Oct 30 13:06:09.506261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.506239    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.537030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.536972    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:06:09.537253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.537040    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:06:09.542746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.542710    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"]
Oct 30 13:06:09.557173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.557131    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.557429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.557336    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.557429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.557384    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg67c\" (UniqueName: \"kubernetes.io/projected/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-kube-api-access-wg67c\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.577353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.577236    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:06:09.577353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.577315    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:06:09.659449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.659388    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.659975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.659524    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.659975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.659570    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg67c\" (UniqueName: \"kubernetes.io/projected/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-kube-api-access-wg67c\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.660199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.660174    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.662456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.662430    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.683560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.683513    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg67c\" (UniqueName: \"kubernetes.io/projected/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-kube-api-access-wg67c\") pod \"maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:09.822354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:09.822227    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:10.197242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:10.196171    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"]
Oct 30 13:06:10.222453 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:06:10.203988    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a0f2fb_5ec0_46aa_8a9e_9b93b276dba0.slice/crio-f4a218d32d8b147761f1537720978047fe46cda133294633bab6f7be08cd88e2 WatchSource:0}: Error finding container f4a218d32d8b147761f1537720978047fe46cda133294633bab6f7be08cd88e2: Status 404 returned error can't find the container with id f4a218d32d8b147761f1537720978047fe46cda133294633bab6f7be08cd88e2
Oct 30 13:06:10.325300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:10.318902    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m" event={"ID":"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0","Type":"ContainerStarted","Data":"f4a218d32d8b147761f1537720978047fe46cda133294633bab6f7be08cd88e2"}
Oct 30 13:06:10.870089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:10.870009    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:06:10.870089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:10.870064    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 13:06:11.329361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:11.329325    4993 generic.go:356] "Generic (PLEG): container finished" podID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerID="3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8" exitCode=0
Oct 30 13:06:11.329594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:11.329395    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" event={"ID":"0cbe7dff-ab80-448b-b0d2-076e434a5cfd","Type":"ContainerDied","Data":"3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8"}
Oct 30 13:06:11.329978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:11.329957    4993 scope.go:122] "RemoveContainer" containerID="3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8"
Oct 30 13:06:11.331443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:11.331408    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m" event={"ID":"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0","Type":"ContainerStarted","Data":"c54d6e69d42d9d3bcd7fea62e634206cf600166ea364440799fc6df8ab702a1c"}
Oct 30 13:06:11.331872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:11.331843    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:11.331954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:11.331925    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:11.332012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:11.331971    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:11.427805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:11.427735    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m" podStartSLOduration=2.427717156 podStartE2EDuration="2.427717156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:11.425511476 +0000 UTC m=+2035.901231088" watchObservedRunningTime="2026-10-30 13:06:11.427717156 +0000 UTC m=+2035.903436766"
Oct 30 13:06:12.345296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:12.345250    4993 generic.go:356] "Generic (PLEG): container finished" podID="58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0" containerID="c54d6e69d42d9d3bcd7fea62e634206cf600166ea364440799fc6df8ab702a1c" exitCode=0
Oct 30 13:06:12.346156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:12.345324    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m" event={"ID":"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0","Type":"ContainerDied","Data":"c54d6e69d42d9d3bcd7fea62e634206cf600166ea364440799fc6df8ab702a1c"}
Oct 30 13:06:12.351836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:12.351803    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" event={"ID":"0cbe7dff-ab80-448b-b0d2-076e434a5cfd","Type":"ContainerStarted","Data":"a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691"}
Oct 30 13:06:13.306219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.306172    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:06:13.306438 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.306414    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:06:13.306521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.306468    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 13:06:13.593605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.593529    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:13.749297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.749264    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-openshift-ca\") pod \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") "
Oct 30 13:06:13.749471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.749307    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-kube-api-access-wg67c\" (UniqueName: \"kubernetes.io/projected/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-kube-api-access-wg67c\") pod \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") "
Oct 30 13:06:13.749471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.749373    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-xks-ca\") pod \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\" (UID: \"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0\") "
Oct 30 13:06:13.749699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.749672    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-openshift-ca" pod "58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0" (UID: "58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:13.750682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.750156    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:13.751644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.751622    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-kube-api-access-wg67c" pod "58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0" (UID: "58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0"). InnerVolumeSpecName "kube-api-access-wg67c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:13.751918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.751889    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-xks-ca" pod "58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0" (UID: "58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:13.851301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.851193    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:13.851301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:13.851251    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg67c\" (UniqueName: \"kubernetes.io/projected/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0-kube-api-access-wg67c\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.307224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:14.307174    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:06:14.307453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:14.307255    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 13:06:14.409984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:14.409922    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"
Oct 30 13:06:14.409984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:14.409956    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m" event={"ID":"58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0","Type":"ContainerDied","Data":"f4a218d32d8b147761f1537720978047fe46cda133294633bab6f7be08cd88e2"}
Oct 30 13:06:14.410316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:14.410007    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4a218d32d8b147761f1537720978047fe46cda133294633bab6f7be08cd88e2"
Oct 30 13:06:15.306538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:15.306505    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:06:15.306962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:15.306556    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 13:06:16.307779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:16.306269    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:06:16.307779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:16.306329    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 13:06:16.446258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:16.446195    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"]
Oct 30 13:06:16.478975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:16.478933    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-405eb1-wwvxr"]
Oct 30 13:06:17.307047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:17.307011    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:06:17.307287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:17.307080    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 13:06:17.594300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:17.594174    4993 scope.go:122] "RemoveContainer" containerID="40f01d08517b43620d2a2ea8bf366344c27cd58e257e51324b1466cf6447f827"
Oct 30 13:06:17.608833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:17.608805    4993 scope.go:122] "RemoveContainer" containerID="cf8ba0a37d1b8dab30b1a8f75f20f6de6758fe58fe860a8d8bd202977fdbb470"
Oct 30 13:06:17.619588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:17.619561    4993 scope.go:122] "RemoveContainer" containerID="25042409dbae88c007502abf189a5df013dcb98944653135abd6e6345c4ccbb9"
Oct 30 13:06:17.695391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:17.695348    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce974da-ff6a-4e52-97ed-ef34f5a7ff36" path="/var/lib/kubelet/pods/7ce974da-ff6a-4e52-97ed-ef34f5a7ff36/volumes"
Oct 30 13:06:18.315636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:18.315603    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:06:18.319281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:18.319254    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:06:18.344682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:18.344641    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:06:18.761514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:18.761461    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"]
Oct 30 13:06:18.767279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:18.767241    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3a236d-w5rr9"]
Oct 30 13:06:19.694571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:19.694535    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca9db10-2972-4c80-bf24-037b01f7f2fd" path="/var/lib/kubelet/pods/dca9db10-2972-4c80-bf24-037b01f7f2fd/volumes"
Oct 30 13:06:22.483610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:22.483571    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"]
Oct 30 13:06:22.487320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:22.487290    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a1ee32-7442m"]
Oct 30 13:06:23.695312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:23.695270    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0" path="/var/lib/kubelet/pods/58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0/volumes"
Oct 30 13:06:28.028172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.028139    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"]
Oct 30 13:06:28.028824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.028783    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy" containerID="cri-o://a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d" gracePeriod=30
Oct 30 13:06:28.912526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.912491    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"]
Oct 30 13:06:28.913033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.913021    4993 cpu_manager.go:414] "containerMap: removing container" podUID="58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0" containerName="revoke-keys"
Oct 30 13:06:28.913092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.913035    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0" containerName="revoke-keys"
Oct 30 13:06:28.913163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.913154    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a0f2fb-5ec0-46aa-8a9e-9b93b276dba0" containerName="revoke-keys"
Oct 30 13:06:28.917191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.917170    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:28.934736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.934712    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:06:28.934894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.934823    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:06:28.939529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.939505    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"]
Oct 30 13:06:28.946173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.946154    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:06:28.946314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:28.946181    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:06:29.087947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.087904    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.088495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.088030    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.088495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.088057    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9x42\" (UniqueName: \"kubernetes.io/projected/1d62190d-c310-4c40-934f-9588ce1ead04-kube-api-access-j9x42\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.189763 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.189670    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.189945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.189784    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.189945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.189810    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9x42\" (UniqueName: \"kubernetes.io/projected/1d62190d-c310-4c40-934f-9588ce1ead04-kube-api-access-j9x42\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.190663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.190632    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.192249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.192193    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.221279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.221195    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9x42\" (UniqueName: \"kubernetes.io/projected/1d62190d-c310-4c40-934f-9588ce1ead04-kube-api-access-j9x42\") pod \"maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.233867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.230073    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:29.660345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.658987    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"]
Oct 30 13:06:29.761331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:29.761283    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv" event={"ID":"1d62190d-c310-4c40-934f-9588ce1ead04","Type":"ContainerStarted","Data":"200de2e3122a4f3c99ed79354e1b53abbe066521ebe99f09b85df752371c4fc7"}
Oct 30 13:06:30.815229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:30.813969    4993 generic.go:356] "Generic (PLEG): container finished" podID="1d62190d-c310-4c40-934f-9588ce1ead04" containerID="d733c7603098e83dcb757ff1b58a646551768e76c4310cf13a39bb4c4b946df9" exitCode=0
Oct 30 13:06:30.815229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:30.814131    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv" event={"ID":"1d62190d-c310-4c40-934f-9588ce1ead04","Type":"ContainerDied","Data":"d733c7603098e83dcb757ff1b58a646551768e76c4310cf13a39bb4c4b946df9"}
Oct 30 13:06:32.008015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.007971    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:32.179894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.179789    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d62190d-c310-4c40-934f-9588ce1ead04-kube-api-access-j9x42\" (UniqueName: \"kubernetes.io/projected/1d62190d-c310-4c40-934f-9588ce1ead04-kube-api-access-j9x42\") pod \"1d62190d-c310-4c40-934f-9588ce1ead04\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") "
Oct 30 13:06:32.180088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.179944    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-xks-ca\") pod \"1d62190d-c310-4c40-934f-9588ce1ead04\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") "
Oct 30 13:06:32.180088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.179968    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-openshift-ca\") pod \"1d62190d-c310-4c40-934f-9588ce1ead04\" (UID: \"1d62190d-c310-4c40-934f-9588ce1ead04\") "
Oct 30 13:06:32.180498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.180468    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-openshift-ca" pod "1d62190d-c310-4c40-934f-9588ce1ead04" (UID: "1d62190d-c310-4c40-934f-9588ce1ead04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:32.182744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.182704    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d62190d-c310-4c40-934f-9588ce1ead04-kube-api-access-j9x42" pod "1d62190d-c310-4c40-934f-9588ce1ead04" (UID: "1d62190d-c310-4c40-934f-9588ce1ead04"). InnerVolumeSpecName "kube-api-access-j9x42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:32.182980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.182955    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-xks-ca" pod "1d62190d-c310-4c40-934f-9588ce1ead04" (UID: "1d62190d-c310-4c40-934f-9588ce1ead04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:32.282451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.282365    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:32.282451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.282408    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d62190d-c310-4c40-934f-9588ce1ead04-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:32.282451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.282425    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9x42\" (UniqueName: \"kubernetes.io/projected/1d62190d-c310-4c40-934f-9588ce1ead04-kube-api-access-j9x42\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:32.836325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.836268    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv" event={"ID":"1d62190d-c310-4c40-934f-9588ce1ead04","Type":"ContainerDied","Data":"200de2e3122a4f3c99ed79354e1b53abbe066521ebe99f09b85df752371c4fc7"}
Oct 30 13:06:32.836510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.836347    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="200de2e3122a4f3c99ed79354e1b53abbe066521ebe99f09b85df752371c4fc7"
Oct 30 13:06:32.836510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:32.836352    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"
Oct 30 13:06:33.402720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.402638    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:06:33.533786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.533752    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-kube-api-access-7fjcf\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-kube-api-access-7fjcf\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.533984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.533815    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-token\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-token\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.533984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.533865    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istiod-ca-cert\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.533984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.533950    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-socket\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.534167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.533986    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-podinfo\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.534167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534036    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-certs\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.534167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534089    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-data\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.534167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534126    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-credential-socket\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.534409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534317    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-envoy\") pod \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\" (UID: \"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b\") "
Oct 30 13:06:33.534466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534400    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-certs" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:33.534835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534575    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-socket" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:33.534835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534761    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-data" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:33.534835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534794    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istiod-ca-cert" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:33.535065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.534912    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-credential-socket" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:33.535065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.535031    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.535065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.535049    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.535065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.535063    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.535356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.535077    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.535356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.535091    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.536759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.536734    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-kube-api-access-7fjcf" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "kube-api-access-7fjcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:33.536859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.536767    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-token" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:33.537265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.537127    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-podinfo" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:33.537265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.537241    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-envoy" pod "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" (UID: "42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:33.640894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.639780    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.640894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.639821    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fjcf\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-kube-api-access-7fjcf\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.640894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.639838    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.640894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.639853    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:06:33.922248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.921458    4993 generic.go:356] "Generic (PLEG): container finished" podID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerID="a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d" exitCode=0
Oct 30 13:06:33.922248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.921661    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" event={"ID":"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b","Type":"ContainerDied","Data":"a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d"}
Oct 30 13:06:33.922248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.921751    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp" event={"ID":"42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b","Type":"ContainerDied","Data":"b39ca3872b04efbc8245213bb8ef7cf547bd4e0d83d2b50b3a02a74bd106374b"}
Oct 30 13:06:33.922248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.921781    4993 scope.go:122] "RemoveContainer" containerID="a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d"
Oct 30 13:06:33.922248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.921959    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"
Oct 30 13:06:33.934808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.934787    4993 scope.go:122] "RemoveContainer" containerID="bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020"
Oct 30 13:06:33.939277 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:06:33.939242    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ce1e86_cc1a_49bc_a3fe_cfe88b20fe9b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ce1e86_cc1a_49bc_a3fe_cfe88b20fe9b.slice/crio-b39ca3872b04efbc8245213bb8ef7cf547bd4e0d83d2b50b3a02a74bd106374b\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:33.939397 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:06:33.939236    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ce1e86_cc1a_49bc_a3fe_cfe88b20fe9b.slice/crio-b39ca3872b04efbc8245213bb8ef7cf547bd4e0d83d2b50b3a02a74bd106374b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ce1e86_cc1a_49bc_a3fe_cfe88b20fe9b.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:06:33.945693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.945668    4993 scope.go:122] "RemoveContainer" containerID="a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d"
Oct 30 13:06:33.945993 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:06:33.945968    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d\": container with ID starting with a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d not found: ID does not exist" containerID="a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d"
Oct 30 13:06:33.946089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.946001    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d"} err="failed to get container status \"a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d\": rpc error: code = NotFound desc = could not find container \"a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d\": container with ID starting with a7249938cd4edaf9197a70363cf3b8eaf8cc79a242ac895a2a3563217dc3521d not found: ID does not exist"
Oct 30 13:06:33.946089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.946019    4993 scope.go:122] "RemoveContainer" containerID="bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020"
Oct 30 13:06:33.946321 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:06:33.946301    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020\": container with ID starting with bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020 not found: ID does not exist" containerID="bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020"
Oct 30 13:06:33.946398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.946331    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020"} err="failed to get container status \"bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020\": rpc error: code = NotFound desc = could not find container \"bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020\": container with ID starting with bcdae6bda6234054fba022555d2e9087e69169d4a41dae5660478eff35232020 not found: ID does not exist"
Oct 30 13:06:33.991951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:33.991913    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"]
Oct 30 13:06:34.004476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:34.004436    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3a236d-openshift-default-75957d8c95-f6czp"]
Oct 30 13:06:35.694843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:35.694813    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" path="/var/lib/kubelet/pods/42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b/volumes"
Oct 30 13:06:38.572354 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:06:38.572312    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:06:47.101662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:47.101623    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"]
Oct 30 13:06:47.114524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:47.114491    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df169d4d-9m22c"]
Oct 30 13:06:47.693384 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:47.693346    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2c86e9-738a-4873-9325-e7d3503676d2" path="/var/lib/kubelet/pods/cb2c86e9-738a-4873-9325-e7d3503676d2/volumes"
Oct 30 13:06:53.046026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:53.045984    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"]
Oct 30 13:06:53.059343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:53.059307    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5066e7be-mqtkx"]
Oct 30 13:06:53.692707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:06:53.692668    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc" path="/var/lib/kubelet/pods/e8987d74-4eaf-4b4b-9028-9a13ea8c6dfc/volumes"
Oct 30 13:07:08.576620 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:07:08.576575    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:07:15.997179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:15.997145    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:07:16.007191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:16.005085    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:07:17.668150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:17.668111    4993 scope.go:122] "RemoveContainer" containerID="d1375fc6723bd879bd2fb8cd9c30a59f259c4d5efa650cfe20b8ff46f06ed0a9"
Oct 30 13:07:17.681970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:17.681943    4993 scope.go:122] "RemoveContainer" containerID="aa855d2ede28fdfcd75266efd8df8ad3973a466bbdea7ff5dfb4a5891c96ea6c"
Oct 30 13:07:29.041744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:29.041702    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"]
Oct 30 13:07:29.048217 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:29.048176    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a121dd-qs8lv"]
Oct 30 13:07:29.692697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:29.692654    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d62190d-c310-4c40-934f-9588ce1ead04" path="/var/lib/kubelet/pods/1d62190d-c310-4c40-934f-9588ce1ead04/volumes"
Oct 30 13:07:38.578707 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:07:38.578657    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:07:49.504835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.504793    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"]
Oct 30 13:07:49.505718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.505693    4993 cpu_manager.go:414] "containerMap: removing container" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy"
Oct 30 13:07:49.505824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.505720    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy"
Oct 30 13:07:49.505824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.505799    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1d62190d-c310-4c40-934f-9588ce1ead04" containerName="revoke-keys"
Oct 30 13:07:49.505824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.505808    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d62190d-c310-4c40-934f-9588ce1ead04" containerName="revoke-keys"
Oct 30 13:07:49.505996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.505950    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy"
Oct 30 13:07:49.505996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.505969    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d62190d-c310-4c40-934f-9588ce1ead04" containerName="revoke-keys"
Oct 30 13:07:49.509702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.509678    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.529964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.529924    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"]
Oct 30 13:07:49.539558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.539517    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:07:49.539751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.539655    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:07:49.613227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.610541    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:07:49.621226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.620973    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b249f\" (UniqueName: \"kubernetes.io/projected/f9bb0534-7d13-473b-baab-6121e5a32b2f-kube-api-access-b249f\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.621226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.621030    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.621226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.621195    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.623184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.621621    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:07:49.722557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.722521    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b249f\" (UniqueName: \"kubernetes.io/projected/f9bb0534-7d13-473b-baab-6121e5a32b2f-kube-api-access-b249f\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.722781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.722564    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.722781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.722700    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.723485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.723462    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.725188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.725163    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.748169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.748138    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b249f\" (UniqueName: \"kubernetes.io/projected/f9bb0534-7d13-473b-baab-6121e5a32b2f-kube-api-access-b249f\") pod \"maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:49.823968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:49.823878    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:50.133891 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:07:50.133855    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bb0534_7d13_473b_baab_6121e5a32b2f.slice/crio-9636d94d84fe51d24b68a481abbf60dbfb59790e1d893369e62dec087e4ca06b WatchSource:0}: Error finding container 9636d94d84fe51d24b68a481abbf60dbfb59790e1d893369e62dec087e4ca06b: Status 404 returned error can't find the container with id 9636d94d84fe51d24b68a481abbf60dbfb59790e1d893369e62dec087e4ca06b
Oct 30 13:07:50.135977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:50.135953    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"]
Oct 30 13:07:51.046736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:51.046694    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs" event={"ID":"f9bb0534-7d13-473b-baab-6121e5a32b2f","Type":"ContainerStarted","Data":"1f9214caa07893c72c6894bedfc41e50ab0d27c5350164724ef4aff44db98ab8"}
Oct 30 13:07:51.047158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:51.046762    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs" event={"ID":"f9bb0534-7d13-473b-baab-6121e5a32b2f","Type":"ContainerStarted","Data":"9636d94d84fe51d24b68a481abbf60dbfb59790e1d893369e62dec087e4ca06b"}
Oct 30 13:07:51.093164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:51.093105    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs" podStartSLOduration=2.093087475 podStartE2EDuration="2.093087475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:51.090047784 +0000 UTC m=+2135.565767400" watchObservedRunningTime="2026-10-30 13:07:51.093087475 +0000 UTC m=+2135.568807085"
Oct 30 13:07:52.056233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:52.056160    4993 generic.go:356] "Generic (PLEG): container finished" podID="f9bb0534-7d13-473b-baab-6121e5a32b2f" containerID="1f9214caa07893c72c6894bedfc41e50ab0d27c5350164724ef4aff44db98ab8" exitCode=0
Oct 30 13:07:52.056233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:52.056216    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs" event={"ID":"f9bb0534-7d13-473b-baab-6121e5a32b2f","Type":"ContainerDied","Data":"1f9214caa07893c72c6894bedfc41e50ab0d27c5350164724ef4aff44db98ab8"}
Oct 30 13:07:53.255593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.255563    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:53.282872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.282837    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-xks-ca\") pod \"f9bb0534-7d13-473b-baab-6121e5a32b2f\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") "
Oct 30 13:07:53.283056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.282982    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bb0534-7d13-473b-baab-6121e5a32b2f-kube-api-access-b249f\" (UniqueName: \"kubernetes.io/projected/f9bb0534-7d13-473b-baab-6121e5a32b2f-kube-api-access-b249f\") pod \"f9bb0534-7d13-473b-baab-6121e5a32b2f\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") "
Oct 30 13:07:53.283128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.283112    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-openshift-ca\") pod \"f9bb0534-7d13-473b-baab-6121e5a32b2f\" (UID: \"f9bb0534-7d13-473b-baab-6121e5a32b2f\") "
Oct 30 13:07:53.283573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.283535    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-openshift-ca" pod "f9bb0534-7d13-473b-baab-6121e5a32b2f" (UID: "f9bb0534-7d13-473b-baab-6121e5a32b2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:53.283787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.283767    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:07:53.285200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.285169    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bb0534-7d13-473b-baab-6121e5a32b2f-kube-api-access-b249f" pod "f9bb0534-7d13-473b-baab-6121e5a32b2f" (UID: "f9bb0534-7d13-473b-baab-6121e5a32b2f"). InnerVolumeSpecName "kube-api-access-b249f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:53.285386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.285366    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-xks-ca" pod "f9bb0534-7d13-473b-baab-6121e5a32b2f" (UID: "f9bb0534-7d13-473b-baab-6121e5a32b2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:53.384976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.384886    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bb0534-7d13-473b-baab-6121e5a32b2f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:07:53.384976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:53.384919    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b249f\" (UniqueName: \"kubernetes.io/projected/f9bb0534-7d13-473b-baab-6121e5a32b2f-kube-api-access-b249f\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:07:54.074884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:54.074853    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"
Oct 30 13:07:54.075104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:54.074887    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs" event={"ID":"f9bb0534-7d13-473b-baab-6121e5a32b2f","Type":"ContainerDied","Data":"9636d94d84fe51d24b68a481abbf60dbfb59790e1d893369e62dec087e4ca06b"}
Oct 30 13:07:54.075104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:07:54.074946    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9636d94d84fe51d24b68a481abbf60dbfb59790e1d893369e62dec087e4ca06b"
Oct 30 13:08:08.575462 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:08:08.575410    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:08:16.124672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:16.124629    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"]
Oct 30 13:08:16.137807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:16.137769    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fa855c-44qzs"]
Oct 30 13:08:17.720223 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:17.720132    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bb0534-7d13-473b-baab-6121e5a32b2f" path="/var/lib/kubelet/pods/f9bb0534-7d13-473b-baab-6121e5a32b2f/volumes"
Oct 30 13:08:17.753101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:17.752272    4993 scope.go:122] "RemoveContainer" containerID="6dc71a4cebc6c0e462bd610ea92b881415419cb47155741566e274a8ed8f5f25"
Oct 30 13:08:17.767281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:17.767250    4993 scope.go:122] "RemoveContainer" containerID="470bfdfe8d2277ec9a622bb4090bd279b140f4ad7551c8d990c9186abf0de110"
Oct 30 13:08:17.780236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:17.780195    4993 scope.go:122] "RemoveContainer" containerID="8c75a4b88303a3ac72e869061b29915dfed77a3056d2df6c2c1b05c42bb50cd6"
Oct 30 13:08:17.795039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:17.795003    4993 scope.go:122] "RemoveContainer" containerID="29247c441f0772718306ab97dc85d5142d7830a805e16d032fbec10ce7dbbe92"
Oct 30 13:08:17.811245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:17.811193    4993 scope.go:122] "RemoveContainer" containerID="e3b3c67b9b6023622081fdf08a94f9fd3406b5db4790786c14a10138563e081b"
Oct 30 13:08:17.822919 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:17.822895    4993 scope.go:122] "RemoveContainer" containerID="70177dc58ab4151d643f03bc7341faf3e6dd07c911e1d57a4d69c2ae873c85ad"
Oct 30 13:08:23.236545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.236499    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"]
Oct 30 13:08:23.237340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.237317    4993 cpu_manager.go:414] "containerMap: removing container" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy"
Oct 30 13:08:23.237453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.237340    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy"
Oct 30 13:08:23.237453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.237374    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f9bb0534-7d13-473b-baab-6121e5a32b2f" containerName="revoke-keys"
Oct 30 13:08:23.237453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.237382    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bb0534-7d13-473b-baab-6121e5a32b2f" containerName="revoke-keys"
Oct 30 13:08:23.237624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.237609    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bb0534-7d13-473b-baab-6121e5a32b2f" containerName="revoke-keys"
Oct 30 13:08:23.237688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.237631    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ce1e86-cc1a-49bc-a3fe-cfe88b20fe9b" containerName="istio-proxy"
Oct 30 13:08:23.242292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.242267    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.262494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.262462    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"]
Oct 30 13:08:23.272267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.272232    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:08:23.272534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.272240    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:08:23.289023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.288987    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxnkx\" (UniqueName: \"kubernetes.io/projected/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-kube-api-access-pxnkx\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.289256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.289060    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.289256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.289084    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.342180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.342142    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:08:23.342446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.342350    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:08:23.390133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.390091    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxnkx\" (UniqueName: \"kubernetes.io/projected/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-kube-api-access-pxnkx\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.390399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.390159    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.390399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.390186    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.390925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.390903    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.392707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.392683    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.416084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.416045    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxnkx\" (UniqueName: \"kubernetes.io/projected/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-kube-api-access-pxnkx\") pod \"maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.557929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.557839    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:23.879494 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:08:23.875922    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422cf708_0a90_4cfe_bd73_c0e9bebe7af6.slice/crio-55e0f9faaea64c06da9efd3c7975c94f183490f87a4c42e3c2769c6787164851 WatchSource:0}: Error finding container 55e0f9faaea64c06da9efd3c7975c94f183490f87a4c42e3c2769c6787164851: Status 404 returned error can't find the container with id 55e0f9faaea64c06da9efd3c7975c94f183490f87a4c42e3c2769c6787164851
Oct 30 13:08:23.879494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:23.876401    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"]
Oct 30 13:08:24.147770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.147680    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"]
Oct 30 13:08:24.148201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.148170    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy" containerID="cri-o://c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba" gracePeriod=30
Oct 30 13:08:24.430308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.430223    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz" event={"ID":"422cf708-0a90-4cfe-bd73-c0e9bebe7af6","Type":"ContainerStarted","Data":"55e0f9faaea64c06da9efd3c7975c94f183490f87a4c42e3c2769c6787164851"}
Oct 30 13:08:24.784703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.784655    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"]
Oct 30 13:08:24.788698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.788670    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.814793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.814674    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"]
Oct 30 13:08:24.867827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.867778    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvbg\" (UniqueName: \"kubernetes.io/projected/fee4aaa2-5da2-43ea-88a8-455c1c396f15-kube-api-access-7lvbg\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.868032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.867919    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.868032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.867961    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.969457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.969418    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvbg\" (UniqueName: \"kubernetes.io/projected/fee4aaa2-5da2-43ea-88a8-455c1c396f15-kube-api-access-7lvbg\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.969660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.969594    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.969660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.969627    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.970545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.970518    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.972344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.972311    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:24.995946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:24.995912    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvbg\" (UniqueName: \"kubernetes.io/projected/fee4aaa2-5da2-43ea-88a8-455c1c396f15-kube-api-access-7lvbg\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:25.104959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:25.104868    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:25.416515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:25.416360    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"]
Oct 30 13:08:25.420881 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:08:25.420847    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee4aaa2_5da2_43ea_88a8_455c1c396f15.slice/crio-cc109c32753d2d71f4b1d1c37d33d3ce855b06a71b63c2b53d5f8de28561f7b8 WatchSource:0}: Error finding container cc109c32753d2d71f4b1d1c37d33d3ce855b06a71b63c2b53d5f8de28561f7b8: Status 404 returned error can't find the container with id cc109c32753d2d71f4b1d1c37d33d3ce855b06a71b63c2b53d5f8de28561f7b8
Oct 30 13:08:25.445178 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:25.445125    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht" event={"ID":"fee4aaa2-5da2-43ea-88a8-455c1c396f15","Type":"ContainerStarted","Data":"cc109c32753d2d71f4b1d1c37d33d3ce855b06a71b63c2b53d5f8de28561f7b8"}
Oct 30 13:08:25.447926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:25.447884    4993 generic.go:356] "Generic (PLEG): container finished" podID="422cf708-0a90-4cfe-bd73-c0e9bebe7af6" containerID="c39e510d3f5edd4cc31f37d08e8fa6f228a2f729168c5c9dc23d764f9f619a8a" exitCode=0
Oct 30 13:08:25.448159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:25.448122    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz" event={"ID":"422cf708-0a90-4cfe-bd73-c0e9bebe7af6","Type":"ContainerDied","Data":"c39e510d3f5edd4cc31f37d08e8fa6f228a2f729168c5c9dc23d764f9f619a8a"}
Oct 30 13:08:26.460558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.460516    4993 generic.go:356] "Generic (PLEG): container finished" podID="fee4aaa2-5da2-43ea-88a8-455c1c396f15" containerID="9fcac392cc0d7ab9c9e05d4448d4ef4de078e366742d76fd1640cecbee2d2e9f" exitCode=0
Oct 30 13:08:26.461097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.460591    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht" event={"ID":"fee4aaa2-5da2-43ea-88a8-455c1c396f15","Type":"ContainerDied","Data":"9fcac392cc0d7ab9c9e05d4448d4ef4de078e366742d76fd1640cecbee2d2e9f"}
Oct 30 13:08:26.634912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.634836    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:26.692846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.692803    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-kube-api-access-pxnkx\" (UniqueName: \"kubernetes.io/projected/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-kube-api-access-pxnkx\") pod \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") "
Oct 30 13:08:26.693045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.692938    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-openshift-ca\") pod \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") "
Oct 30 13:08:26.693154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.693134    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-xks-ca\") pod \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\" (UID: \"422cf708-0a90-4cfe-bd73-c0e9bebe7af6\") "
Oct 30 13:08:26.693343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.693312    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-openshift-ca" pod "422cf708-0a90-4cfe-bd73-c0e9bebe7af6" (UID: "422cf708-0a90-4cfe-bd73-c0e9bebe7af6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:26.693670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.693651    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:26.695387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.695359    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-xks-ca" pod "422cf708-0a90-4cfe-bd73-c0e9bebe7af6" (UID: "422cf708-0a90-4cfe-bd73-c0e9bebe7af6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:26.696024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.695675    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-kube-api-access-pxnkx" pod "422cf708-0a90-4cfe-bd73-c0e9bebe7af6" (UID: "422cf708-0a90-4cfe-bd73-c0e9bebe7af6"). InnerVolumeSpecName "kube-api-access-pxnkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:26.794835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.794799    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:26.794835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:26.794833    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxnkx\" (UniqueName: \"kubernetes.io/projected/422cf708-0a90-4cfe-bd73-c0e9bebe7af6-kube-api-access-pxnkx\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:27.466441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.466399    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"]
Oct 30 13:08:27.467746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.467717    4993 cpu_manager.go:414] "containerMap: removing container" podUID="422cf708-0a90-4cfe-bd73-c0e9bebe7af6" containerName="revoke-keys"
Oct 30 13:08:27.467746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.467746    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422cf708-0a90-4cfe-bd73-c0e9bebe7af6" containerName="revoke-keys"
Oct 30 13:08:27.467946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.467913    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="422cf708-0a90-4cfe-bd73-c0e9bebe7af6" containerName="revoke-keys"
Oct 30 13:08:27.471499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.471474    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.473812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.473777    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"
Oct 30 13:08:27.473934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.473813    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz" event={"ID":"422cf708-0a90-4cfe-bd73-c0e9bebe7af6","Type":"ContainerDied","Data":"55e0f9faaea64c06da9efd3c7975c94f183490f87a4c42e3c2769c6787164851"}
Oct 30 13:08:27.473934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.473868    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e0f9faaea64c06da9efd3c7975c94f183490f87a4c42e3c2769c6787164851"
Oct 30 13:08:27.490031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.489788    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"]
Oct 30 13:08:27.606907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.606870    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swswt\" (UniqueName: \"kubernetes.io/projected/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-kube-api-access-swswt\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.607074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.606921    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.607216 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.607180    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.672671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.672226    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:27.708470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.708438    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swswt\" (UniqueName: \"kubernetes.io/projected/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-kube-api-access-swswt\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.708640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.708482    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.708732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.708713    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.709269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.709246    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.711126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.711107    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.741853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.741817    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swswt\" (UniqueName: \"kubernetes.io/projected/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-kube-api-access-swswt\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.807464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.807415    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:27.809743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.809719    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee4aaa2-5da2-43ea-88a8-455c1c396f15-kube-api-access-7lvbg\" (UniqueName: \"kubernetes.io/projected/fee4aaa2-5da2-43ea-88a8-455c1c396f15-kube-api-access-7lvbg\") pod \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") "
Oct 30 13:08:27.809947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.809929    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-openshift-ca\") pod \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") "
Oct 30 13:08:27.810024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.809985    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-xks-ca\") pod \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\" (UID: \"fee4aaa2-5da2-43ea-88a8-455c1c396f15\") "
Oct 30 13:08:27.810406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.810327    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-openshift-ca" pod "fee4aaa2-5da2-43ea-88a8-455c1c396f15" (UID: "fee4aaa2-5da2-43ea-88a8-455c1c396f15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:27.810690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.810673    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:27.812053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.812031    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee4aaa2-5da2-43ea-88a8-455c1c396f15-kube-api-access-7lvbg" pod "fee4aaa2-5da2-43ea-88a8-455c1c396f15" (UID: "fee4aaa2-5da2-43ea-88a8-455c1c396f15"). InnerVolumeSpecName "kube-api-access-7lvbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:27.812551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.812526    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-xks-ca" pod "fee4aaa2-5da2-43ea-88a8-455c1c396f15" (UID: "fee4aaa2-5da2-43ea-88a8-455c1c396f15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:27.911722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.911683    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lvbg\" (UniqueName: \"kubernetes.io/projected/fee4aaa2-5da2-43ea-88a8-455c1c396f15-kube-api-access-7lvbg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:27.911722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:27.911720    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee4aaa2-5da2-43ea-88a8-455c1c396f15-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:28.122964 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:08:28.122918    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9941ebd_2433_4a4b_8f05_0615c51f6ff7.slice/crio-ccb758ee11ccf810b759cdf08a664ad8b6783d6b9779c4c274e3bd6a0590f723 WatchSource:0}: Error finding container ccb758ee11ccf810b759cdf08a664ad8b6783d6b9779c4c274e3bd6a0590f723: Status 404 returned error can't find the container with id ccb758ee11ccf810b759cdf08a664ad8b6783d6b9779c4c274e3bd6a0590f723
Oct 30 13:08:28.123743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:28.123712    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"]
Oct 30 13:08:28.484910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:28.484874    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"
Oct 30 13:08:28.485407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:28.484902    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht" event={"ID":"fee4aaa2-5da2-43ea-88a8-455c1c396f15","Type":"ContainerDied","Data":"cc109c32753d2d71f4b1d1c37d33d3ce855b06a71b63c2b53d5f8de28561f7b8"}
Oct 30 13:08:28.485407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:28.484964    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc109c32753d2d71f4b1d1c37d33d3ce855b06a71b63c2b53d5f8de28561f7b8"
Oct 30 13:08:28.487164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:28.487127    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq" event={"ID":"a9941ebd-2433-4a4b-8f05-0615c51f6ff7","Type":"ContainerStarted","Data":"dbfdd85cb83656ab1acba5d4f561a575f667a61f0eebebd581a3d49d84d1bd2b"}
Oct 30 13:08:28.487399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:28.487196    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq" event={"ID":"a9941ebd-2433-4a4b-8f05-0615c51f6ff7","Type":"ContainerStarted","Data":"ccb758ee11ccf810b759cdf08a664ad8b6783d6b9779c4c274e3bd6a0590f723"}
Oct 30 13:08:28.537036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:28.536981    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq" podStartSLOduration=1.53696687 podStartE2EDuration="1.53696687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:28.5346354 +0000 UTC m=+2173.010355009" watchObservedRunningTime="2026-10-30 13:08:28.53696687 +0000 UTC m=+2173.012686503"
Oct 30 13:08:29.449581 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.449546    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:08:29.497765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.497727    4993 generic.go:356] "Generic (PLEG): container finished" podID="a9941ebd-2433-4a4b-8f05-0615c51f6ff7" containerID="dbfdd85cb83656ab1acba5d4f561a575f667a61f0eebebd581a3d49d84d1bd2b" exitCode=0
Oct 30 13:08:29.498319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.497813    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq" event={"ID":"a9941ebd-2433-4a4b-8f05-0615c51f6ff7","Type":"ContainerDied","Data":"dbfdd85cb83656ab1acba5d4f561a575f667a61f0eebebd581a3d49d84d1bd2b"}
Oct 30 13:08:29.499933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.499899    4993 generic.go:356] "Generic (PLEG): container finished" podID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerID="c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba" exitCode=0
Oct 30 13:08:29.500065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.499946    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"
Oct 30 13:08:29.500065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.499988    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" event={"ID":"b9a0d1be-3eed-44bd-8358-e091469a84ea","Type":"ContainerDied","Data":"c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba"}
Oct 30 13:08:29.500065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.500054    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8" event={"ID":"b9a0d1be-3eed-44bd-8358-e091469a84ea","Type":"ContainerDied","Data":"c6fc33c156da7d70ccbed7f09d1ae1990a63a8c50c2bbe82bb44b7430f4723fd"}
Oct 30 13:08:29.500177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.500071    4993 scope.go:122] "RemoveContainer" containerID="c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba"
Oct 30 13:08:29.510989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.510969    4993 scope.go:122] "RemoveContainer" containerID="749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076"
Oct 30 13:08:29.522378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.522347    4993 scope.go:122] "RemoveContainer" containerID="c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba"
Oct 30 13:08:29.522754 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:08:29.522724    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba\": container with ID starting with c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba not found: ID does not exist" containerID="c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba"
Oct 30 13:08:29.522849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.522766    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba"} err="failed to get container status \"c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba\": rpc error: code = NotFound desc = could not find container \"c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba\": container with ID starting with c07f33af32de4fd82d1f3ab2f62e57d66510029530f0ac7af9291c327f62adba not found: ID does not exist"
Oct 30 13:08:29.522849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.522794    4993 scope.go:122] "RemoveContainer" containerID="749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076"
Oct 30 13:08:29.523098 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:08:29.523075    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076\": container with ID starting with 749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076 not found: ID does not exist" containerID="749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076"
Oct 30 13:08:29.523156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.523103    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076"} err="failed to get container status \"749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076\": rpc error: code = NotFound desc = could not find container \"749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076\": container with ID starting with 749f709fa1fad4352d3eb28a319898d72490caad0ab1447ea385577fcc610076 not found: ID does not exist"
Oct 30 13:08:29.532505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532188    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-kube-api-access-rx7f9\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-kube-api-access-rx7f9\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.532505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532359    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-data\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.532505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532467    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-socket\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.532786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532582    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a0d1be-3eed-44bd-8358-e091469a84ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a0d1be-3eed-44bd-8358-e091469a84ea-istiod-ca-cert\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.532786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532700    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-certs\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.532786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532767    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-token\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-token\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.533305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532860    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-credential-socket\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.533305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532919    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-envoy\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.533305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.532985    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-podinfo\") pod \"b9a0d1be-3eed-44bd-8358-e091469a84ea\" (UID: \"b9a0d1be-3eed-44bd-8358-e091469a84ea\") "
Oct 30 13:08:29.533771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.533536    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-data" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:29.534110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.533910    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-socket" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:29.534291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.534263    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-credential-socket" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:29.542271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.542168    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a0d1be-3eed-44bd-8358-e091469a84ea-istiod-ca-cert" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:29.556189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.548375    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-envoy" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:29.556189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.551359    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-certs" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:29.563518 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.563473    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-kube-api-access-rx7f9" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "kube-api-access-rx7f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:29.563871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.563489    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-token" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:29.569846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.567311    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-podinfo" pod "b9a0d1be-3eed-44bd-8358-e091469a84ea" (UID: "b9a0d1be-3eed-44bd-8358-e091469a84ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:08:29.634536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634494    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a0d1be-3eed-44bd-8358-e091469a84ea-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.634536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634524    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.634536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634534    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.634536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634542    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.634902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634551    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.634902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634560    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.634902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634568    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx7f9\" (UniqueName: \"kubernetes.io/projected/b9a0d1be-3eed-44bd-8358-e091469a84ea-kube-api-access-rx7f9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.634902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634576    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.634902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.634584    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a0d1be-3eed-44bd-8358-e091469a84ea-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:29.851538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.851499    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"]
Oct 30 13:08:29.858671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:29.858627    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa855c-openshift-default-675f746795-9hqn8"]
Oct 30 13:08:30.678333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.678307    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:30.746863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.746826    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-xks-ca\") pod \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") "
Oct 30 13:08:30.747020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.746920    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-openshift-ca\") pod \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") "
Oct 30 13:08:30.747098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.747022    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-kube-api-access-swswt\" (UniqueName: \"kubernetes.io/projected/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-kube-api-access-swswt\") pod \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\" (UID: \"a9941ebd-2433-4a4b-8f05-0615c51f6ff7\") "
Oct 30 13:08:30.747445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.747403    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-openshift-ca" pod "a9941ebd-2433-4a4b-8f05-0615c51f6ff7" (UID: "a9941ebd-2433-4a4b-8f05-0615c51f6ff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:30.747577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.747553    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:30.749678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.749652    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-xks-ca" pod "a9941ebd-2433-4a4b-8f05-0615c51f6ff7" (UID: "a9941ebd-2433-4a4b-8f05-0615c51f6ff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:30.749764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.749687    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-kube-api-access-swswt" pod "a9941ebd-2433-4a4b-8f05-0615c51f6ff7" (UID: "a9941ebd-2433-4a4b-8f05-0615c51f6ff7"). InnerVolumeSpecName "kube-api-access-swswt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:30.848783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.848737    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:30.848783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:30.848775    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swswt\" (UniqueName: \"kubernetes.io/projected/a9941ebd-2433-4a4b-8f05-0615c51f6ff7-kube-api-access-swswt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:31.517463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:31.517433    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"
Oct 30 13:08:31.517662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:31.517431    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq" event={"ID":"a9941ebd-2433-4a4b-8f05-0615c51f6ff7","Type":"ContainerDied","Data":"ccb758ee11ccf810b759cdf08a664ad8b6783d6b9779c4c274e3bd6a0590f723"}
Oct 30 13:08:31.517662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:31.517595    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb758ee11ccf810b759cdf08a664ad8b6783d6b9779c4c274e3bd6a0590f723"
Oct 30 13:08:31.693769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:31.693733    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" path="/var/lib/kubelet/pods/b9a0d1be-3eed-44bd-8358-e091469a84ea/volumes"
Oct 30 13:08:38.631618 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:08:38.631577    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:08:40.076697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:40.076659    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"]
Oct 30 13:08:40.092115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:40.092068    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c35df-txqcq"]
Oct 30 13:08:41.752174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:41.752129    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9941ebd-2433-4a4b-8f05-0615c51f6ff7" path="/var/lib/kubelet/pods/a9941ebd-2433-4a4b-8f05-0615c51f6ff7/volumes"
Oct 30 13:08:45.063086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:45.063044    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"]
Oct 30 13:08:45.075912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:45.075864    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e0e74-fl8ht"]
Oct 30 13:08:45.693333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:45.693285    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee4aaa2-5da2-43ea-88a8-455c1c396f15" path="/var/lib/kubelet/pods/fee4aaa2-5da2-43ea-88a8-455c1c396f15/volumes"
Oct 30 13:08:50.457675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.457624    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"]
Oct 30 13:08:50.458505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458478    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a9941ebd-2433-4a4b-8f05-0615c51f6ff7" containerName="revoke-keys"
Oct 30 13:08:50.458505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458505    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9941ebd-2433-4a4b-8f05-0615c51f6ff7" containerName="revoke-keys"
Oct 30 13:08:50.458686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458545    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy"
Oct 30 13:08:50.458686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458553    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy"
Oct 30 13:08:50.458686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458589    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fee4aaa2-5da2-43ea-88a8-455c1c396f15" containerName="revoke-keys"
Oct 30 13:08:50.458686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458598    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee4aaa2-5da2-43ea-88a8-455c1c396f15" containerName="revoke-keys"
Oct 30 13:08:50.458686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458613    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy"
Oct 30 13:08:50.458686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458620    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy"
Oct 30 13:08:50.459037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458763    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9941ebd-2433-4a4b-8f05-0615c51f6ff7" containerName="revoke-keys"
Oct 30 13:08:50.459037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458780    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy"
Oct 30 13:08:50.459037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.458795    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee4aaa2-5da2-43ea-88a8-455c1c396f15" containerName="revoke-keys"
Oct 30 13:08:50.464265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.464230    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.481778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.481741    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"]
Oct 30 13:08:50.490715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.490679    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:08:50.491257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.491143    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:08:50.516917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.516880    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:08:50.517239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.517014    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:08:50.535922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.535867    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.536120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.535938    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.536120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.536096    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbh4q\" (UniqueName: \"kubernetes.io/projected/8a4b12ed-e080-4204-8078-3f118a9a118c-kube-api-access-gbh4q\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.637840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.637801    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.638031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.637848    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.638031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.637911    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbh4q\" (UniqueName: \"kubernetes.io/projected/8a4b12ed-e080-4204-8078-3f118a9a118c-kube-api-access-gbh4q\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.638553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.638531    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.640473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.640449    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.663770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.663731    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbh4q\" (UniqueName: \"kubernetes.io/projected/8a4b12ed-e080-4204-8078-3f118a9a118c-kube-api-access-gbh4q\") pod \"maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:50.780649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:50.780598    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:51.090053 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:08:51.090005    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4b12ed_e080_4204_8078_3f118a9a118c.slice/crio-73052214dd233ea3ea676ad79bb58bdc9d156c13c873636020400da132a4e27b WatchSource:0}: Error finding container 73052214dd233ea3ea676ad79bb58bdc9d156c13c873636020400da132a4e27b: Status 404 returned error can't find the container with id 73052214dd233ea3ea676ad79bb58bdc9d156c13c873636020400da132a4e27b
Oct 30 13:08:51.091154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:51.091110    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"]
Oct 30 13:08:51.894115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:51.894065    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p" event={"ID":"8a4b12ed-e080-4204-8078-3f118a9a118c","Type":"ContainerStarted","Data":"5bea54239b20473c901b5c468096bb0910686ccd9400aca1d3dd242d48252346"}
Oct 30 13:08:51.894683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:51.894142    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p" event={"ID":"8a4b12ed-e080-4204-8078-3f118a9a118c","Type":"ContainerStarted","Data":"73052214dd233ea3ea676ad79bb58bdc9d156c13c873636020400da132a4e27b"}
Oct 30 13:08:51.951462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:51.951388    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p" podStartSLOduration=1.9513693189999999 podStartE2EDuration="1.951369319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:51.948168541 +0000 UTC m=+2196.423888151" watchObservedRunningTime="2026-10-30 13:08:51.951369319 +0000 UTC m=+2196.427088929"
Oct 30 13:08:52.907685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:52.907648    4993 generic.go:356] "Generic (PLEG): container finished" podID="8a4b12ed-e080-4204-8078-3f118a9a118c" containerID="5bea54239b20473c901b5c468096bb0910686ccd9400aca1d3dd242d48252346" exitCode=0
Oct 30 13:08:52.908094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:52.907729    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p" event={"ID":"8a4b12ed-e080-4204-8078-3f118a9a118c","Type":"ContainerDied","Data":"5bea54239b20473c901b5c468096bb0910686ccd9400aca1d3dd242d48252346"}
Oct 30 13:08:54.289230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.285068    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:54.460379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.458682    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-openshift-ca\") pod \"8a4b12ed-e080-4204-8078-3f118a9a118c\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") "
Oct 30 13:08:54.460379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.458859    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-xks-ca\") pod \"8a4b12ed-e080-4204-8078-3f118a9a118c\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") "
Oct 30 13:08:54.460379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.458943    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4b12ed-e080-4204-8078-3f118a9a118c-kube-api-access-gbh4q\" (UniqueName: \"kubernetes.io/projected/8a4b12ed-e080-4204-8078-3f118a9a118c-kube-api-access-gbh4q\") pod \"8a4b12ed-e080-4204-8078-3f118a9a118c\" (UID: \"8a4b12ed-e080-4204-8078-3f118a9a118c\") "
Oct 30 13:08:54.460845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.460792    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-openshift-ca" pod "8a4b12ed-e080-4204-8078-3f118a9a118c" (UID: "8a4b12ed-e080-4204-8078-3f118a9a118c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:54.486827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.486755    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-xks-ca" pod "8a4b12ed-e080-4204-8078-3f118a9a118c" (UID: "8a4b12ed-e080-4204-8078-3f118a9a118c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:54.489791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.489749    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4b12ed-e080-4204-8078-3f118a9a118c-kube-api-access-gbh4q" pod "8a4b12ed-e080-4204-8078-3f118a9a118c" (UID: "8a4b12ed-e080-4204-8078-3f118a9a118c"). InnerVolumeSpecName "kube-api-access-gbh4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:54.548045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.548003    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"]
Oct 30 13:08:54.557134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.557092    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4a97e9-mvllz"]
Oct 30 13:08:54.567339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.560918    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:54.567339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.561011    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4b12ed-e080-4204-8078-3f118a9a118c-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:54.567339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.561050    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbh4q\" (UniqueName: \"kubernetes.io/projected/8a4b12ed-e080-4204-8078-3f118a9a118c-kube-api-access-gbh4q\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:08:54.926310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.926278    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"
Oct 30 13:08:54.926502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.926282    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p" event={"ID":"8a4b12ed-e080-4204-8078-3f118a9a118c","Type":"ContainerDied","Data":"73052214dd233ea3ea676ad79bb58bdc9d156c13c873636020400da132a4e27b"}
Oct 30 13:08:54.926502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:54.926407    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73052214dd233ea3ea676ad79bb58bdc9d156c13c873636020400da132a4e27b"
Oct 30 13:08:55.695925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:08:55.695879    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422cf708-0a90-4cfe-bd73-c0e9bebe7af6" path="/var/lib/kubelet/pods/422cf708-0a90-4cfe-bd73-c0e9bebe7af6/volumes"
Oct 30 13:09:08.571171 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:09:08.571062    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:09:11.040874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.040832    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"]
Oct 30 13:09:11.041753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.041725    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8a4b12ed-e080-4204-8078-3f118a9a118c" containerName="revoke-keys"
Oct 30 13:09:11.041944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.041923    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4b12ed-e080-4204-8078-3f118a9a118c" containerName="revoke-keys"
Oct 30 13:09:11.042310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.042291    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a0d1be-3eed-44bd-8358-e091469a84ea" containerName="istio-proxy"
Oct 30 13:09:11.042461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.042449    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4b12ed-e080-4204-8078-3f118a9a118c" containerName="revoke-keys"
Oct 30 13:09:11.047386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.047342    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.068793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.068757    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"]
Oct 30 13:09:11.078731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.078703    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:11.079952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.079746    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:09:11.080166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.079894    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:09:11.171364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.168531    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:09:11.334294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.310406    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4b2b\" (UniqueName: \"kubernetes.io/projected/e12a8870-3857-46d7-9816-daa5e109e395-kube-api-access-l4b2b\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.334294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.310563    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.334294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.310795    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.411778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.411720    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.412001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.411805    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4b2b\" (UniqueName: \"kubernetes.io/projected/e12a8870-3857-46d7-9816-daa5e109e395-kube-api-access-l4b2b\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.412001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.411870    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.412658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.412632    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.414455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.414418    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.440515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.440484    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4b2b\" (UniqueName: \"kubernetes.io/projected/e12a8870-3857-46d7-9816-daa5e109e395-kube-api-access-l4b2b\") pod \"maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.570123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.570081    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"]
Oct 30 13:09:11.576094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.576057    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-aa466d-4jt9p"]
Oct 30 13:09:11.699020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.698904    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:11.712258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:11.710227    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4b12ed-e080-4204-8078-3f118a9a118c" path="/var/lib/kubelet/pods/8a4b12ed-e080-4204-8078-3f118a9a118c/volumes"
Oct 30 13:09:12.030889 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:09:12.030850    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12a8870_3857_46d7_9816_daa5e109e395.slice/crio-c50d423e961822acbf3feb2bd030a15ccf45d35512149d8409c4583e16b062e0 WatchSource:0}: Error finding container c50d423e961822acbf3feb2bd030a15ccf45d35512149d8409c4583e16b062e0: Status 404 returned error can't find the container with id c50d423e961822acbf3feb2bd030a15ccf45d35512149d8409c4583e16b062e0
Oct 30 13:09:12.033131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:12.033103    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"]
Oct 30 13:09:12.143739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:12.143704    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5" event={"ID":"e12a8870-3857-46d7-9816-daa5e109e395","Type":"ContainerStarted","Data":"c50d423e961822acbf3feb2bd030a15ccf45d35512149d8409c4583e16b062e0"}
Oct 30 13:09:13.152539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:13.152497    4993 generic.go:356] "Generic (PLEG): container finished" podID="e12a8870-3857-46d7-9816-daa5e109e395" containerID="e6b0cfe4ec6b253d6e9eaf53fc2a849f0bc8d23101090cb58acbd929eb732bd9" exitCode=0
Oct 30 13:09:13.152967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:13.152562    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5" event={"ID":"e12a8870-3857-46d7-9816-daa5e109e395","Type":"ContainerDied","Data":"e6b0cfe4ec6b253d6e9eaf53fc2a849f0bc8d23101090cb58acbd929eb732bd9"}
Oct 30 13:09:14.357931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.357896    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:14.459726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.459615    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-xks-ca\") pod \"e12a8870-3857-46d7-9816-daa5e109e395\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") "
Oct 30 13:09:14.459901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.459779    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-openshift-ca\") pod \"e12a8870-3857-46d7-9816-daa5e109e395\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") "
Oct 30 13:09:14.459901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.459833    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12a8870-3857-46d7-9816-daa5e109e395-kube-api-access-l4b2b\" (UniqueName: \"kubernetes.io/projected/e12a8870-3857-46d7-9816-daa5e109e395-kube-api-access-l4b2b\") pod \"e12a8870-3857-46d7-9816-daa5e109e395\" (UID: \"e12a8870-3857-46d7-9816-daa5e109e395\") "
Oct 30 13:09:14.460277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.460192    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-openshift-ca" pod "e12a8870-3857-46d7-9816-daa5e109e395" (UID: "e12a8870-3857-46d7-9816-daa5e109e395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:14.460496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.460474    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:14.462078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.462047    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-xks-ca" pod "e12a8870-3857-46d7-9816-daa5e109e395" (UID: "e12a8870-3857-46d7-9816-daa5e109e395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:14.462255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.462227    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12a8870-3857-46d7-9816-daa5e109e395-kube-api-access-l4b2b" pod "e12a8870-3857-46d7-9816-daa5e109e395" (UID: "e12a8870-3857-46d7-9816-daa5e109e395"). InnerVolumeSpecName "kube-api-access-l4b2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:14.562088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.562042    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4b2b\" (UniqueName: \"kubernetes.io/projected/e12a8870-3857-46d7-9816-daa5e109e395-kube-api-access-l4b2b\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:14.562088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:14.562073    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12a8870-3857-46d7-9816-daa5e109e395-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:15.169725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:15.169689    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"
Oct 30 13:09:15.169916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:15.169721    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5" event={"ID":"e12a8870-3857-46d7-9816-daa5e109e395","Type":"ContainerDied","Data":"c50d423e961822acbf3feb2bd030a15ccf45d35512149d8409c4583e16b062e0"}
Oct 30 13:09:15.169916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:15.169774    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c50d423e961822acbf3feb2bd030a15ccf45d35512149d8409c4583e16b062e0"
Oct 30 13:09:17.883900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:17.883861    4993 scope.go:122] "RemoveContainer" containerID="2299aa33da0cd9d7dd6697ceb3aefa0b9c80f0dafa4fc48b8116209665225b58"
Oct 30 13:09:17.894762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:17.894735    4993 scope.go:122] "RemoveContainer" containerID="f6da0bd4deeae888c7e7e2e0e3a4fd33a51c5d232d3842ba0b1c9b6c36d5266f"
Oct 30 13:09:24.309727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.309684    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"]
Oct 30 13:09:24.310711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.310689    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e12a8870-3857-46d7-9816-daa5e109e395" containerName="revoke-keys"
Oct 30 13:09:24.310767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.310713    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12a8870-3857-46d7-9816-daa5e109e395" containerName="revoke-keys"
Oct 30 13:09:24.310931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.310918    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12a8870-3857-46d7-9816-daa5e109e395" containerName="revoke-keys"
Oct 30 13:09:24.317289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.317263    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.342502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.342460    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7gfn\" (UniqueName: \"kubernetes.io/projected/21397549-3a57-492e-9efe-8cfc9664fbb9-kube-api-access-l7gfn\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.342713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.342520    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.342766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.342727    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.343809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.343779    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"]
Oct 30 13:09:24.362567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.362533    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:09:24.362794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.362679    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:09:24.362794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.362733    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:24.362922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.362800    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:09:24.444191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.444148    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.444416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.444336    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7gfn\" (UniqueName: \"kubernetes.io/projected/21397549-3a57-492e-9efe-8cfc9664fbb9-kube-api-access-l7gfn\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.444416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.444389    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.445154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.445125    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.446760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.446741    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.470150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.470111    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7gfn\" (UniqueName: \"kubernetes.io/projected/21397549-3a57-492e-9efe-8cfc9664fbb9-kube-api-access-l7gfn\") pod \"maas-api-revoke-sub-556b7370-wpd88\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.640699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.636723    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:24.974535 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:09:24.974501    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21397549_3a57_492e_9efe_8cfc9664fbb9.slice/crio-4634303bcdb4510c86f11464d4268e18af7f43bed53f5866edd5b8fb37182bb7 WatchSource:0}: Error finding container 4634303bcdb4510c86f11464d4268e18af7f43bed53f5866edd5b8fb37182bb7: Status 404 returned error can't find the container with id 4634303bcdb4510c86f11464d4268e18af7f43bed53f5866edd5b8fb37182bb7
Oct 30 13:09:24.975748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:24.975720    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"]
Oct 30 13:09:25.292883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:25.292830    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88" event={"ID":"21397549-3a57-492e-9efe-8cfc9664fbb9","Type":"ContainerStarted","Data":"4634303bcdb4510c86f11464d4268e18af7f43bed53f5866edd5b8fb37182bb7"}
Oct 30 13:09:26.305853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:26.305814    4993 generic.go:356] "Generic (PLEG): container finished" podID="21397549-3a57-492e-9efe-8cfc9664fbb9" containerID="648752ec6e9c23dc18fe8e6fa47eb5910359f72abb687b3899bafb5e9023a03f" exitCode=0
Oct 30 13:09:26.306287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:26.305905    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88" event={"ID":"21397549-3a57-492e-9efe-8cfc9664fbb9","Type":"ContainerDied","Data":"648752ec6e9c23dc18fe8e6fa47eb5910359f72abb687b3899bafb5e9023a03f"}
Oct 30 13:09:27.500362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.500326    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:27.620557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.620452    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-xks-ca\") pod \"21397549-3a57-492e-9efe-8cfc9664fbb9\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") "
Oct 30 13:09:27.620721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.620567    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21397549-3a57-492e-9efe-8cfc9664fbb9-kube-api-access-l7gfn\" (UniqueName: \"kubernetes.io/projected/21397549-3a57-492e-9efe-8cfc9664fbb9-kube-api-access-l7gfn\") pod \"21397549-3a57-492e-9efe-8cfc9664fbb9\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") "
Oct 30 13:09:27.620721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.620700    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-openshift-ca\") pod \"21397549-3a57-492e-9efe-8cfc9664fbb9\" (UID: \"21397549-3a57-492e-9efe-8cfc9664fbb9\") "
Oct 30 13:09:27.621087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.621051    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-openshift-ca" pod "21397549-3a57-492e-9efe-8cfc9664fbb9" (UID: "21397549-3a57-492e-9efe-8cfc9664fbb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:27.621196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.621161    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:27.622818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.622781    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-xks-ca" pod "21397549-3a57-492e-9efe-8cfc9664fbb9" (UID: "21397549-3a57-492e-9efe-8cfc9664fbb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:27.622818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.622797    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21397549-3a57-492e-9efe-8cfc9664fbb9-kube-api-access-l7gfn" pod "21397549-3a57-492e-9efe-8cfc9664fbb9" (UID: "21397549-3a57-492e-9efe-8cfc9664fbb9"). InnerVolumeSpecName "kube-api-access-l7gfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:27.723022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.722975    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7gfn\" (UniqueName: \"kubernetes.io/projected/21397549-3a57-492e-9efe-8cfc9664fbb9-kube-api-access-l7gfn\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:27.723022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:27.723021    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21397549-3a57-492e-9efe-8cfc9664fbb9-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:28.321903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:28.321863    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"]
Oct 30 13:09:28.325183 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:28.325155    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"
Oct 30 13:09:28.325368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:28.325182    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88" event={"ID":"21397549-3a57-492e-9efe-8cfc9664fbb9","Type":"ContainerDied","Data":"4634303bcdb4510c86f11464d4268e18af7f43bed53f5866edd5b8fb37182bb7"}
Oct 30 13:09:28.325368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:28.325262    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4634303bcdb4510c86f11464d4268e18af7f43bed53f5866edd5b8fb37182bb7"
Oct 30 13:09:28.334576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:28.334536    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e954cd-lqqn5"]
Oct 30 13:09:29.693399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:29.693356    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12a8870-3857-46d7-9816-daa5e109e395" path="/var/lib/kubelet/pods/e12a8870-3857-46d7-9816-daa5e109e395/volumes"
Oct 30 13:09:30.243507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.243460    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"]
Oct 30 13:09:30.244020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.244006    4993 cpu_manager.go:414] "containerMap: removing container" podUID="21397549-3a57-492e-9efe-8cfc9664fbb9" containerName="revoke-keys"
Oct 30 13:09:30.244072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.244021    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21397549-3a57-492e-9efe-8cfc9664fbb9" containerName="revoke-keys"
Oct 30 13:09:30.244142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.244132    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="21397549-3a57-492e-9efe-8cfc9664fbb9" containerName="revoke-keys"
Oct 30 13:09:30.248100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.248075    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.270505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.270468    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"]
Oct 30 13:09:30.278914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.278882    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:09:30.279116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.278981    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:30.279116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.279046    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:09:30.281526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.281495    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:09:30.400755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.400707    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.400984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.400766    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.400984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.400905    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/18989a21-9740-4fbd-8440-24d8ab33615c-kube-api-access-btrjt\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.501916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.501803    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.501916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.501872    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.502128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.501924    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/18989a21-9740-4fbd-8440-24d8ab33615c-kube-api-access-btrjt\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.502745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.502703    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.504615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.504591    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.531952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.531913    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/18989a21-9740-4fbd-8440-24d8ab33615c-kube-api-access-btrjt\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.567911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.567866    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:30.905258 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:09:30.905188    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18989a21_9740_4fbd_8440_24d8ab33615c.slice/crio-97ae9feba238533ad816398c34c553c504614f0235135666dbd6589c1e050661 WatchSource:0}: Error finding container 97ae9feba238533ad816398c34c553c504614f0235135666dbd6589c1e050661: Status 404 returned error can't find the container with id 97ae9feba238533ad816398c34c553c504614f0235135666dbd6589c1e050661
Oct 30 13:09:30.906759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:30.906728    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"]
Oct 30 13:09:31.353085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:31.353049    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n" event={"ID":"18989a21-9740-4fbd-8440-24d8ab33615c","Type":"ContainerStarted","Data":"be8db447dead6de4a515a5ecbd2d211870027a77ff254039fb7e3170fd03f6cf"}
Oct 30 13:09:31.353343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:31.353112    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n" event={"ID":"18989a21-9740-4fbd-8440-24d8ab33615c","Type":"ContainerStarted","Data":"97ae9feba238533ad816398c34c553c504614f0235135666dbd6589c1e050661"}
Oct 30 13:09:31.399187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:31.399110    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n" podStartSLOduration=1.399092576 podStartE2EDuration="1.399092576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:31.396930271 +0000 UTC m=+2235.872649881" watchObservedRunningTime="2026-10-30 13:09:31.399092576 +0000 UTC m=+2235.874812186"
Oct 30 13:09:32.364609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:32.364560    4993 generic.go:356] "Generic (PLEG): container finished" podID="18989a21-9740-4fbd-8440-24d8ab33615c" containerID="be8db447dead6de4a515a5ecbd2d211870027a77ff254039fb7e3170fd03f6cf" exitCode=0
Oct 30 13:09:32.365087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:32.364642    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n" event={"ID":"18989a21-9740-4fbd-8440-24d8ab33615c","Type":"ContainerDied","Data":"be8db447dead6de4a515a5ecbd2d211870027a77ff254039fb7e3170fd03f6cf"}
Oct 30 13:09:33.548102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.548075    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:33.562080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.562042    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-openshift-ca\") pod \"18989a21-9740-4fbd-8440-24d8ab33615c\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") "
Oct 30 13:09:33.562302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.562110    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18989a21-9740-4fbd-8440-24d8ab33615c-kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/18989a21-9740-4fbd-8440-24d8ab33615c-kube-api-access-btrjt\") pod \"18989a21-9740-4fbd-8440-24d8ab33615c\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") "
Oct 30 13:09:33.562302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.562251    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-xks-ca\") pod \"18989a21-9740-4fbd-8440-24d8ab33615c\" (UID: \"18989a21-9740-4fbd-8440-24d8ab33615c\") "
Oct 30 13:09:33.562465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.562437    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-openshift-ca" pod "18989a21-9740-4fbd-8440-24d8ab33615c" (UID: "18989a21-9740-4fbd-8440-24d8ab33615c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:33.562765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.562750    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.564504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.564477    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-xks-ca" pod "18989a21-9740-4fbd-8440-24d8ab33615c" (UID: "18989a21-9740-4fbd-8440-24d8ab33615c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:33.564622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.564501    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18989a21-9740-4fbd-8440-24d8ab33615c-kube-api-access-btrjt" pod "18989a21-9740-4fbd-8440-24d8ab33615c" (UID: "18989a21-9740-4fbd-8440-24d8ab33615c"). InnerVolumeSpecName "kube-api-access-btrjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:33.664560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.664455    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/18989a21-9740-4fbd-8440-24d8ab33615c-kube-api-access-btrjt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:33.664560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:33.664491    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18989a21-9740-4fbd-8440-24d8ab33615c-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:34.389279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:34.389241    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"
Oct 30 13:09:34.389279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:34.389250    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n" event={"ID":"18989a21-9740-4fbd-8440-24d8ab33615c","Type":"ContainerDied","Data":"97ae9feba238533ad816398c34c553c504614f0235135666dbd6589c1e050661"}
Oct 30 13:09:34.389497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:34.389312    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ae9feba238533ad816398c34c553c504614f0235135666dbd6589c1e050661"
Oct 30 13:09:38.247949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.247908    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"]
Oct 30 13:09:38.248532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.248516    4993 cpu_manager.go:414] "containerMap: removing container" podUID="18989a21-9740-4fbd-8440-24d8ab33615c" containerName="revoke-keys"
Oct 30 13:09:38.248532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.248533    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18989a21-9740-4fbd-8440-24d8ab33615c" containerName="revoke-keys"
Oct 30 13:09:38.248660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.248651    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="18989a21-9740-4fbd-8440-24d8ab33615c" containerName="revoke-keys"
Oct 30 13:09:38.252447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.252418    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.278628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.277856    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"]
Oct 30 13:09:38.285900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.285864    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:09:38.286096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.285965    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:09:38.286096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.286002    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:38.286096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.286039    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:09:38.337332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.337291    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.337570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.337456    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.337570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.337492    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwck\" (UniqueName: \"kubernetes.io/projected/97671825-1e33-49cf-abc1-3c1e25f64259-kube-api-access-lvwck\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.438821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.438777    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.438821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.438821    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwck\" (UniqueName: \"kubernetes.io/projected/97671825-1e33-49cf-abc1-3c1e25f64259-kube-api-access-lvwck\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.439109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.439035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.439669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.439643    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.441699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.441669    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.466295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.466261    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwck\" (UniqueName: \"kubernetes.io/projected/97671825-1e33-49cf-abc1-3c1e25f64259-kube-api-access-lvwck\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.570444 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:09:38.570352    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:09:38.571625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.571602    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:38.889769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:38.889724    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"]
Oct 30 13:09:39.289290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.289239    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"]
Oct 30 13:09:39.293539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.293504    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.314603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.314565    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"]
Oct 30 13:09:39.353550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.353516    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85skp\" (UniqueName: \"kubernetes.io/projected/8cd737ed-3c76-458d-afb1-7aba8e386892-kube-api-access-85skp\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.353767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.353583    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.353767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.353749    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.433881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.433840    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx" event={"ID":"97671825-1e33-49cf-abc1-3c1e25f64259","Type":"ContainerStarted","Data":"be7f293614bead2c03d5f0a1bb53edf342ec689f2621e4c43733909cca3812bc"}
Oct 30 13:09:39.434110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.433897    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx" event={"ID":"97671825-1e33-49cf-abc1-3c1e25f64259","Type":"ContainerStarted","Data":"4f37a23640a370c6659dac272f7e2f1f830647e9fa021270b9585c6964e43c32"}
Oct 30 13:09:39.455692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.455645    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85skp\" (UniqueName: \"kubernetes.io/projected/8cd737ed-3c76-458d-afb1-7aba8e386892-kube-api-access-85skp\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.455858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.455750    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.455942 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.455915    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.456545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.456521    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.458292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.458265    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.484014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.483976    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85skp\" (UniqueName: \"kubernetes.io/projected/8cd737ed-3c76-458d-afb1-7aba8e386892-kube-api-access-85skp\") pod \"maas-api-revoke-sub-c7b4490c-28rf9\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.487686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.487630    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx" podStartSLOduration=1.487614007 podStartE2EDuration="1.487614007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:39.485399838 +0000 UTC m=+2243.961119450" watchObservedRunningTime="2026-10-30 13:09:39.487614007 +0000 UTC m=+2243.963333675"
Oct 30 13:09:39.610786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.610696    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:39.928082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:39.928044    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"]
Oct 30 13:09:39.928311 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:09:39.928286    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd737ed_3c76_458d_afb1_7aba8e386892.slice/crio-c31fe34e2a52a43982b4d8db86ef7a66f1a3603efe433c6b5f84005f591c07fa WatchSource:0}: Error finding container c31fe34e2a52a43982b4d8db86ef7a66f1a3603efe433c6b5f84005f591c07fa: Status 404 returned error can't find the container with id c31fe34e2a52a43982b4d8db86ef7a66f1a3603efe433c6b5f84005f591c07fa
Oct 30 13:09:40.446326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:40.446199    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9" event={"ID":"8cd737ed-3c76-458d-afb1-7aba8e386892","Type":"ContainerStarted","Data":"276afc516d838b37c25baca8801e302f4e1dadb6942a109019f8a1898a2e91a8"}
Oct 30 13:09:40.446326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:40.446297    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9" event={"ID":"8cd737ed-3c76-458d-afb1-7aba8e386892","Type":"ContainerStarted","Data":"c31fe34e2a52a43982b4d8db86ef7a66f1a3603efe433c6b5f84005f591c07fa"}
Oct 30 13:09:40.448298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:40.448265    4993 generic.go:356] "Generic (PLEG): container finished" podID="97671825-1e33-49cf-abc1-3c1e25f64259" containerID="be7f293614bead2c03d5f0a1bb53edf342ec689f2621e4c43733909cca3812bc" exitCode=0
Oct 30 13:09:40.448422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:40.448330    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx" event={"ID":"97671825-1e33-49cf-abc1-3c1e25f64259","Type":"ContainerDied","Data":"be7f293614bead2c03d5f0a1bb53edf342ec689f2621e4c43733909cca3812bc"}
Oct 30 13:09:40.501640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:40.501583    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9" podStartSLOduration=1.501570086 podStartE2EDuration="1.501570086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:40.500388042 +0000 UTC m=+2244.976107654" watchObservedRunningTime="2026-10-30 13:09:40.501570086 +0000 UTC m=+2244.977289697"
Oct 30 13:09:41.273358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.273312    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"]
Oct 30 13:09:41.277350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.277322    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.320664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.320617    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"]
Oct 30 13:09:41.386672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.386633    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.386672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.386684    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.386952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.386925    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdtdz\" (UniqueName: \"kubernetes.io/projected/a25607eb-52ce-4e2d-a176-e5f42c1012ae-kube-api-access-qdtdz\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.459698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.459662    4993 generic.go:356] "Generic (PLEG): container finished" podID="8cd737ed-3c76-458d-afb1-7aba8e386892" containerID="276afc516d838b37c25baca8801e302f4e1dadb6942a109019f8a1898a2e91a8" exitCode=0
Oct 30 13:09:41.460260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.459758    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9" event={"ID":"8cd737ed-3c76-458d-afb1-7aba8e386892","Type":"ContainerDied","Data":"276afc516d838b37c25baca8801e302f4e1dadb6942a109019f8a1898a2e91a8"}
Oct 30 13:09:41.488531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.488492    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtdz\" (UniqueName: \"kubernetes.io/projected/a25607eb-52ce-4e2d-a176-e5f42c1012ae-kube-api-access-qdtdz\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.488749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.488624    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.488749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.488658    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.489420 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.489388    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.491780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.491757    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.527248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.526899    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtdz\" (UniqueName: \"kubernetes.io/projected/a25607eb-52ce-4e2d-a176-e5f42c1012ae-kube-api-access-qdtdz\") pod \"maas-api-revoke-sub-ae4ae85f-g9j4t\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.592798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.592757    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:41.663006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.662966    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:41.792776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.792677    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-xks-ca\") pod \"97671825-1e33-49cf-abc1-3c1e25f64259\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") "
Oct 30 13:09:41.792776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.792726    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-openshift-ca\") pod \"97671825-1e33-49cf-abc1-3c1e25f64259\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") "
Oct 30 13:09:41.793024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.792869    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97671825-1e33-49cf-abc1-3c1e25f64259-kube-api-access-lvwck\" (UniqueName: \"kubernetes.io/projected/97671825-1e33-49cf-abc1-3c1e25f64259-kube-api-access-lvwck\") pod \"97671825-1e33-49cf-abc1-3c1e25f64259\" (UID: \"97671825-1e33-49cf-abc1-3c1e25f64259\") "
Oct 30 13:09:41.793254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.793224    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-openshift-ca" pod "97671825-1e33-49cf-abc1-3c1e25f64259" (UID: "97671825-1e33-49cf-abc1-3c1e25f64259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:41.793648 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.793629    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:41.795105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.795071    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-xks-ca" pod "97671825-1e33-49cf-abc1-3c1e25f64259" (UID: "97671825-1e33-49cf-abc1-3c1e25f64259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:41.795197 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.795160    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97671825-1e33-49cf-abc1-3c1e25f64259-kube-api-access-lvwck" pod "97671825-1e33-49cf-abc1-3c1e25f64259" (UID: "97671825-1e33-49cf-abc1-3c1e25f64259"). InnerVolumeSpecName "kube-api-access-lvwck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:41.894934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.894893    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvwck\" (UniqueName: \"kubernetes.io/projected/97671825-1e33-49cf-abc1-3c1e25f64259-kube-api-access-lvwck\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:41.894934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.894933    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97671825-1e33-49cf-abc1-3c1e25f64259-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:41.940294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:41.940252    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"]
Oct 30 13:09:42.469959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.469870    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"
Oct 30 13:09:42.469959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.469895    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx" event={"ID":"97671825-1e33-49cf-abc1-3c1e25f64259","Type":"ContainerDied","Data":"4f37a23640a370c6659dac272f7e2f1f830647e9fa021270b9585c6964e43c32"}
Oct 30 13:09:42.469959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.469951    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f37a23640a370c6659dac272f7e2f1f830647e9fa021270b9585c6964e43c32"
Oct 30 13:09:42.472080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.472040    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t" event={"ID":"a25607eb-52ce-4e2d-a176-e5f42c1012ae","Type":"ContainerStarted","Data":"fec7fe0b4deb0a1d0be0c1e29f2f8c5067fbbc3dd565f5f29fab5e3a5f090760"}
Oct 30 13:09:42.472285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.472110    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t" event={"ID":"a25607eb-52ce-4e2d-a176-e5f42c1012ae","Type":"ContainerStarted","Data":"acb606466fe7f664433098dacc456a8edaa00255c4de06e7c995e47b4977ce75"}
Oct 30 13:09:42.520611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.520548    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t" podStartSLOduration=1.520531157 podStartE2EDuration="1.520531157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:42.517111894 +0000 UTC m=+2246.992831505" watchObservedRunningTime="2026-10-30 13:09:42.520531157 +0000 UTC m=+2246.996250767"
Oct 30 13:09:42.638417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.638389    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:42.706537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.706504    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd737ed-3c76-458d-afb1-7aba8e386892-kube-api-access-85skp\" (UniqueName: \"kubernetes.io/projected/8cd737ed-3c76-458d-afb1-7aba8e386892-kube-api-access-85skp\") pod \"8cd737ed-3c76-458d-afb1-7aba8e386892\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") "
Oct 30 13:09:42.706731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.706697    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-openshift-ca\") pod \"8cd737ed-3c76-458d-afb1-7aba8e386892\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") "
Oct 30 13:09:42.706785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.706760    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-xks-ca\") pod \"8cd737ed-3c76-458d-afb1-7aba8e386892\" (UID: \"8cd737ed-3c76-458d-afb1-7aba8e386892\") "
Oct 30 13:09:42.707167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.707130    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-openshift-ca" pod "8cd737ed-3c76-458d-afb1-7aba8e386892" (UID: "8cd737ed-3c76-458d-afb1-7aba8e386892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:42.707493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.707465    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.709001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.708970    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-xks-ca" pod "8cd737ed-3c76-458d-afb1-7aba8e386892" (UID: "8cd737ed-3c76-458d-afb1-7aba8e386892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:42.709093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.708999    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd737ed-3c76-458d-afb1-7aba8e386892-kube-api-access-85skp" pod "8cd737ed-3c76-458d-afb1-7aba8e386892" (UID: "8cd737ed-3c76-458d-afb1-7aba8e386892"). InnerVolumeSpecName "kube-api-access-85skp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:42.808917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.808877    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85skp\" (UniqueName: \"kubernetes.io/projected/8cd737ed-3c76-458d-afb1-7aba8e386892-kube-api-access-85skp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.808917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:42.808911    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd737ed-3c76-458d-afb1-7aba8e386892-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:43.485125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:43.485092    4993 generic.go:356] "Generic (PLEG): container finished" podID="a25607eb-52ce-4e2d-a176-e5f42c1012ae" containerID="fec7fe0b4deb0a1d0be0c1e29f2f8c5067fbbc3dd565f5f29fab5e3a5f090760" exitCode=0
Oct 30 13:09:43.485686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:43.485180    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t" event={"ID":"a25607eb-52ce-4e2d-a176-e5f42c1012ae","Type":"ContainerDied","Data":"fec7fe0b4deb0a1d0be0c1e29f2f8c5067fbbc3dd565f5f29fab5e3a5f090760"}
Oct 30 13:09:43.486996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:43.486969    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9" event={"ID":"8cd737ed-3c76-458d-afb1-7aba8e386892","Type":"ContainerDied","Data":"c31fe34e2a52a43982b4d8db86ef7a66f1a3603efe433c6b5f84005f591c07fa"}
Oct 30 13:09:43.487123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:43.487005    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"
Oct 30 13:09:43.487123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:43.487021    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31fe34e2a52a43982b4d8db86ef7a66f1a3603efe433c6b5f84005f591c07fa"
Oct 30 13:09:44.686405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.686376    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:44.836709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.836603    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-xks-ca\") pod \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") "
Oct 30 13:09:44.836887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.836819    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-openshift-ca\") pod \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") "
Oct 30 13:09:44.836935 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.836909    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25607eb-52ce-4e2d-a176-e5f42c1012ae-kube-api-access-qdtdz\" (UniqueName: \"kubernetes.io/projected/a25607eb-52ce-4e2d-a176-e5f42c1012ae-kube-api-access-qdtdz\") pod \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\" (UID: \"a25607eb-52ce-4e2d-a176-e5f42c1012ae\") "
Oct 30 13:09:44.837156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.837131    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-openshift-ca" pod "a25607eb-52ce-4e2d-a176-e5f42c1012ae" (UID: "a25607eb-52ce-4e2d-a176-e5f42c1012ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:44.837525 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.837497    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:44.838941 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.838916    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-xks-ca" pod "a25607eb-52ce-4e2d-a176-e5f42c1012ae" (UID: "a25607eb-52ce-4e2d-a176-e5f42c1012ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:44.839045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.839016    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25607eb-52ce-4e2d-a176-e5f42c1012ae-kube-api-access-qdtdz" pod "a25607eb-52ce-4e2d-a176-e5f42c1012ae" (UID: "a25607eb-52ce-4e2d-a176-e5f42c1012ae"). InnerVolumeSpecName "kube-api-access-qdtdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:44.938932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.938890    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdtdz\" (UniqueName: \"kubernetes.io/projected/a25607eb-52ce-4e2d-a176-e5f42c1012ae-kube-api-access-qdtdz\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:44.938932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:44.938929    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25607eb-52ce-4e2d-a176-e5f42c1012ae-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:45.035008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:45.034968    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"]
Oct 30 13:09:45.040314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:45.040272    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b953eea9-cqh44"]
Oct 30 13:09:45.505295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:45.505245    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t" event={"ID":"a25607eb-52ce-4e2d-a176-e5f42c1012ae","Type":"ContainerDied","Data":"acb606466fe7f664433098dacc456a8edaa00255c4de06e7c995e47b4977ce75"}
Oct 30 13:09:45.505486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:45.505309    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acb606466fe7f664433098dacc456a8edaa00255c4de06e7c995e47b4977ce75"
Oct 30 13:09:45.505486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:45.505262    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"
Oct 30 13:09:45.694981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:45.694936    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d8667d-c6b0-4eb1-affe-2e7033a91391" path="/var/lib/kubelet/pods/a6d8667d-c6b0-4eb1-affe-2e7033a91391/volumes"
Oct 30 13:09:52.417234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.417143    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"]
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418103    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a25607eb-52ce-4e2d-a176-e5f42c1012ae" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418138    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25607eb-52ce-4e2d-a176-e5f42c1012ae" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418158    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8cd737ed-3c76-458d-afb1-7aba8e386892" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418166    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd737ed-3c76-458d-afb1-7aba8e386892" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418290    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97671825-1e33-49cf-abc1-3c1e25f64259" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418300    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97671825-1e33-49cf-abc1-3c1e25f64259" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418477    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25607eb-52ce-4e2d-a176-e5f42c1012ae" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418495    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd737ed-3c76-458d-afb1-7aba8e386892" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.418516    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="97671825-1e33-49cf-abc1-3c1e25f64259" containerName="revoke-keys"
Oct 30 13:09:52.428233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.426717    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:52.459231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.457730    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:09:52.459231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.458307    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:09:52.459231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.458367    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:52.459231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.458496    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:09:52.536348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.460035    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"]
Oct 30 13:09:52.583229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.582277    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rm8\" (UniqueName: \"kubernetes.io/projected/ba3490bc-b617-4cb0-a765-efe4d384647f-kube-api-access-52rm8\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:52.650234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.635888    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:52.650234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.635973    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:52.782254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.772941    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rm8\" (UniqueName: \"kubernetes.io/projected/ba3490bc-b617-4cb0-a765-efe4d384647f-kube-api-access-52rm8\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:52.782254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.773069    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:52.782254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.773120    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:52.782254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:52.780854    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:53.203319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:53.194525    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:53.203319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:53.197760    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rm8\" (UniqueName: \"kubernetes.io/projected/ba3490bc-b617-4cb0-a765-efe4d384647f-kube-api-access-52rm8\") pod \"maas-api-revoke-sub-949d6391-xvtfd\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:53.323231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:53.320837    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:53.778151 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:09:53.778112    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3490bc_b617_4cb0_a765_efe4d384647f.slice/crio-26c9f8e0c789a988687b6f886223d3b46159b2ff4e292ab6321ee779a214404c WatchSource:0}: Error finding container 26c9f8e0c789a988687b6f886223d3b46159b2ff4e292ab6321ee779a214404c: Status 404 returned error can't find the container with id 26c9f8e0c789a988687b6f886223d3b46159b2ff4e292ab6321ee779a214404c
Oct 30 13:09:53.778686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:53.778182    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"]
Oct 30 13:09:54.278546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.278278    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"]
Oct 30 13:09:54.283001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.282967    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.316805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.316766    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"]
Oct 30 13:09:54.368043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.367988    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.368264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.368160    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb557\" (UniqueName: \"kubernetes.io/projected/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-kube-api-access-cb557\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.368351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.368324    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.470159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.470047    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.470406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.470264    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb557\" (UniqueName: \"kubernetes.io/projected/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-kube-api-access-cb557\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.470406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.470388    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.477264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.475121    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.477264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.475831    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.502662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.502625    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb557\" (UniqueName: \"kubernetes.io/projected/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-kube-api-access-cb557\") pod \"maas-api-revoke-sub-97b851af-mv8h9\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.586541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.586496    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"]
Oct 30 13:09:54.592059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.592024    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.601540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.601507    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:54.609524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.609485    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"]
Oct 30 13:09:54.673254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.673012    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5tz\" (UniqueName: \"kubernetes.io/projected/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-kube-api-access-bp5tz\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.673254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.673122    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.673254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.673175    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.724890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.724802    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd" event={"ID":"ba3490bc-b617-4cb0-a765-efe4d384647f","Type":"ContainerStarted","Data":"704f91c55f9ada83d5aacc9b47f8a6e17fbc92ee4c27b7d4d33f2b429d5251ba"}
Oct 30 13:09:54.724890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.724865    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd" event={"ID":"ba3490bc-b617-4cb0-a765-efe4d384647f","Type":"ContainerStarted","Data":"26c9f8e0c789a988687b6f886223d3b46159b2ff4e292ab6321ee779a214404c"}
Oct 30 13:09:54.771048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.770990    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd" podStartSLOduration=2.7709757699999997 podStartE2EDuration="2.77097577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:54.768867997 +0000 UTC m=+2259.244587609" watchObservedRunningTime="2026-10-30 13:09:54.77097577 +0000 UTC m=+2259.246695379"
Oct 30 13:09:54.774804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.774772    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5tz\" (UniqueName: \"kubernetes.io/projected/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-kube-api-access-bp5tz\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.774984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.774888    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.774984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.774939    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.776841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.776523    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.779266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.778922    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.800990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.800955    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5tz\" (UniqueName: \"kubernetes.io/projected/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-kube-api-access-bp5tz\") pod \"maas-api-revoke-sub-b11fc584-b7r7c\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.920044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.919997    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:54.932761 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:09:54.932727    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee7dea9_5e02_4a05_aa98_ee0cb578e0a6.slice/crio-e9e23bd98ff8f71a6f3205090a4305ae12e251cff87f517dc911f506b46b64ef WatchSource:0}: Error finding container e9e23bd98ff8f71a6f3205090a4305ae12e251cff87f517dc911f506b46b64ef: Status 404 returned error can't find the container with id e9e23bd98ff8f71a6f3205090a4305ae12e251cff87f517dc911f506b46b64ef
Oct 30 13:09:54.932972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:54.932955    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"]
Oct 30 13:09:55.239284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.239236    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"]
Oct 30 13:09:55.735312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.735268    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c" event={"ID":"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8","Type":"ContainerStarted","Data":"bc256370767595db1d025782266054b089d26d572fbec1ad2da5faebc9f72f7f"}
Oct 30 13:09:55.735527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.735338    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c" event={"ID":"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8","Type":"ContainerStarted","Data":"38e42b28776daa25f243d7f51953bde954f3cd7d6f74777e0502f029b853e0a3"}
Oct 30 13:09:55.737354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.737320    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9" event={"ID":"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6","Type":"ContainerStarted","Data":"4ef97a13544cc67ca7719f3d94f3abff15ab88cb06d70b18160785bf0dbfc982"}
Oct 30 13:09:55.737499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.737373    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9" event={"ID":"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6","Type":"ContainerStarted","Data":"e9e23bd98ff8f71a6f3205090a4305ae12e251cff87f517dc911f506b46b64ef"}
Oct 30 13:09:55.740022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.739994    4993 generic.go:356] "Generic (PLEG): container finished" podID="ba3490bc-b617-4cb0-a765-efe4d384647f" containerID="704f91c55f9ada83d5aacc9b47f8a6e17fbc92ee4c27b7d4d33f2b429d5251ba" exitCode=0
Oct 30 13:09:55.740158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.740062    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd" event={"ID":"ba3490bc-b617-4cb0-a765-efe4d384647f","Type":"ContainerDied","Data":"704f91c55f9ada83d5aacc9b47f8a6e17fbc92ee4c27b7d4d33f2b429d5251ba"}
Oct 30 13:09:55.783826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.783739    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c" podStartSLOduration=1.7837133870000002 podStartE2EDuration="1.783713387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:55.779003119 +0000 UTC m=+2260.254722730" watchObservedRunningTime="2026-10-30 13:09:55.783713387 +0000 UTC m=+2260.259433000"
Oct 30 13:09:55.881975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:55.881903    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9" podStartSLOduration=1.881880369 podStartE2EDuration="1.881880369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:55.87718719 +0000 UTC m=+2260.352906806" watchObservedRunningTime="2026-10-30 13:09:55.881880369 +0000 UTC m=+2260.357599982"
Oct 30 13:09:56.749786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:56.749751    4993 generic.go:356] "Generic (PLEG): container finished" podID="1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6" containerID="4ef97a13544cc67ca7719f3d94f3abff15ab88cb06d70b18160785bf0dbfc982" exitCode=0
Oct 30 13:09:56.749996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:56.749836    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9" event={"ID":"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6","Type":"ContainerDied","Data":"4ef97a13544cc67ca7719f3d94f3abff15ab88cb06d70b18160785bf0dbfc982"}
Oct 30 13:09:56.751655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:56.751626    4993 generic.go:356] "Generic (PLEG): container finished" podID="fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8" containerID="bc256370767595db1d025782266054b089d26d572fbec1ad2da5faebc9f72f7f" exitCode=0
Oct 30 13:09:56.751785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:56.751693    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c" event={"ID":"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8","Type":"ContainerDied","Data":"bc256370767595db1d025782266054b089d26d572fbec1ad2da5faebc9f72f7f"}
Oct 30 13:09:56.965625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:56.965585    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:57.007580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.007470    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-openshift-ca\") pod \"ba3490bc-b617-4cb0-a765-efe4d384647f\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") "
Oct 30 13:09:57.007750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.007684    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3490bc-b617-4cb0-a765-efe4d384647f-kube-api-access-52rm8\" (UniqueName: \"kubernetes.io/projected/ba3490bc-b617-4cb0-a765-efe4d384647f-kube-api-access-52rm8\") pod \"ba3490bc-b617-4cb0-a765-efe4d384647f\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") "
Oct 30 13:09:57.007750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.007735    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-xks-ca\") pod \"ba3490bc-b617-4cb0-a765-efe4d384647f\" (UID: \"ba3490bc-b617-4cb0-a765-efe4d384647f\") "
Oct 30 13:09:57.007942 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.007914    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-openshift-ca" pod "ba3490bc-b617-4cb0-a765-efe4d384647f" (UID: "ba3490bc-b617-4cb0-a765-efe4d384647f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:57.008291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.008264    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:57.009952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.009929    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-xks-ca" pod "ba3490bc-b617-4cb0-a765-efe4d384647f" (UID: "ba3490bc-b617-4cb0-a765-efe4d384647f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:57.010060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.010013    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3490bc-b617-4cb0-a765-efe4d384647f-kube-api-access-52rm8" pod "ba3490bc-b617-4cb0-a765-efe4d384647f" (UID: "ba3490bc-b617-4cb0-a765-efe4d384647f"). InnerVolumeSpecName "kube-api-access-52rm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:57.109448 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.109410    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rm8\" (UniqueName: \"kubernetes.io/projected/ba3490bc-b617-4cb0-a765-efe4d384647f-kube-api-access-52rm8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:57.109448 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.109445    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3490bc-b617-4cb0-a765-efe4d384647f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:57.762477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.762442    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"
Oct 30 13:09:57.762477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.762470    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd" event={"ID":"ba3490bc-b617-4cb0-a765-efe4d384647f","Type":"ContainerDied","Data":"26c9f8e0c789a988687b6f886223d3b46159b2ff4e292ab6321ee779a214404c"}
Oct 30 13:09:57.762767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.762522    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c9f8e0c789a988687b6f886223d3b46159b2ff4e292ab6321ee779a214404c"
Oct 30 13:09:57.944945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.944907    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:57.972694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:57.972665    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:09:58.019723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.019679    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-xks-ca\") pod \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") "
Oct 30 13:09:58.019926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.019733    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-openshift-ca\") pod \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") "
Oct 30 13:09:58.019926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.019780    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-openshift-ca\") pod \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") "
Oct 30 13:09:58.019926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.019812    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-xks-ca\") pod \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") "
Oct 30 13:09:58.019926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.019886    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-kube-api-access-cb557\" (UniqueName: \"kubernetes.io/projected/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-kube-api-access-cb557\") pod \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\" (UID: \"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6\") "
Oct 30 13:09:58.020167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.019983    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-kube-api-access-bp5tz\" (UniqueName: \"kubernetes.io/projected/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-kube-api-access-bp5tz\") pod \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\" (UID: \"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8\") "
Oct 30 13:09:58.020260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.020152    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-openshift-ca" pod "fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8" (UID: "fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:58.020260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.020158    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-openshift-ca" pod "1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6" (UID: "1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:58.020886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.020856    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:58.021028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.020889    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:58.022392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.022356    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-xks-ca" pod "fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8" (UID: "fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:58.022491 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.022437    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-xks-ca" pod "1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6" (UID: "1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:58.022721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.022692    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-kube-api-access-bp5tz" pod "fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8" (UID: "fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8"). InnerVolumeSpecName "kube-api-access-bp5tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:58.022852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.022835    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-kube-api-access-cb557" pod "1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6" (UID: "1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6"). InnerVolumeSpecName "kube-api-access-cb557". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:58.122564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.122518    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:58.122564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.122557    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:58.122564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.122569    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb557\" (UniqueName: \"kubernetes.io/projected/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6-kube-api-access-cb557\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:58.122830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.122582    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5tz\" (UniqueName: \"kubernetes.io/projected/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8-kube-api-access-bp5tz\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:09:58.772511 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.772473    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"
Oct 30 13:09:58.772719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.772508    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c" event={"ID":"fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8","Type":"ContainerDied","Data":"38e42b28776daa25f243d7f51953bde954f3cd7d6f74777e0502f029b853e0a3"}
Oct 30 13:09:58.772719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.772567    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38e42b28776daa25f243d7f51953bde954f3cd7d6f74777e0502f029b853e0a3"
Oct 30 13:09:58.774473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.774436    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9" event={"ID":"1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6","Type":"ContainerDied","Data":"e9e23bd98ff8f71a6f3205090a4305ae12e251cff87f517dc911f506b46b64ef"}
Oct 30 13:09:58.774614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.774495    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e23bd98ff8f71a6f3205090a4305ae12e251cff87f517dc911f506b46b64ef"
Oct 30 13:09:58.774614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:09:58.774568    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"
Oct 30 13:10:03.879020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:03.878981    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"]
Oct 30 13:10:03.892385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:03.892352    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9ea8f-qv5bx"]
Oct 30 13:10:04.808846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.808810    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"]
Oct 30 13:10:04.809404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809388    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8" containerName="revoke-keys"
Oct 30 13:10:04.809482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809405    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8" containerName="revoke-keys"
Oct 30 13:10:04.809482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809427    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ba3490bc-b617-4cb0-a765-efe4d384647f" containerName="revoke-keys"
Oct 30 13:10:04.809482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809432    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3490bc-b617-4cb0-a765-efe4d384647f" containerName="revoke-keys"
Oct 30 13:10:04.809482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809454    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6" containerName="revoke-keys"
Oct 30 13:10:04.809482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809459    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6" containerName="revoke-keys"
Oct 30 13:10:04.809632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809539    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6" containerName="revoke-keys"
Oct 30 13:10:04.809632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809549    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8" containerName="revoke-keys"
Oct 30 13:10:04.809632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.809558    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3490bc-b617-4cb0-a765-efe4d384647f" containerName="revoke-keys"
Oct 30 13:10:04.814217 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.814190    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:04.832372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.832346    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:04.832522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.832456    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:04.852861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.852824    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"]
Oct 30 13:10:04.909914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.909879    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:10:04.910396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.909924    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:04.936569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.936527    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:04.936715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.936680    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2h5z\" (UniqueName: \"kubernetes.io/projected/812684fe-ef9c-42d9-b140-960ad92a954a-kube-api-access-q2h5z\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:04.936775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:04.936729    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:05.038428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.038383    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:05.038637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.038569    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2h5z\" (UniqueName: \"kubernetes.io/projected/812684fe-ef9c-42d9-b140-960ad92a954a-kube-api-access-q2h5z\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:05.038709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.038647    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:05.039339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.039313    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:05.040862 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.040844    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:05.063949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.063883    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2h5z\" (UniqueName: \"kubernetes.io/projected/812684fe-ef9c-42d9-b140-960ad92a954a-kube-api-access-q2h5z\") pod \"maas-api-revoke-sub-90c6a7e3-px5v2\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:05.128134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.128096    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:05.454396 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:05.447281    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812684fe_ef9c_42d9_b140_960ad92a954a.slice/crio-4b8a9c96f60341ea747fd5fd611cfdf3dd0f89de83aa4356e86354ae8300f07a WatchSource:0}: Error finding container 4b8a9c96f60341ea747fd5fd611cfdf3dd0f89de83aa4356e86354ae8300f07a: Status 404 returned error can't find the container with id 4b8a9c96f60341ea747fd5fd611cfdf3dd0f89de83aa4356e86354ae8300f07a
Oct 30 13:10:05.454396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.452573    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"]
Oct 30 13:10:05.692745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.692656    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97671825-1e33-49cf-abc1-3c1e25f64259" path="/var/lib/kubelet/pods/97671825-1e33-49cf-abc1-3c1e25f64259/volumes"
Oct 30 13:10:05.835287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.835243    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2" event={"ID":"812684fe-ef9c-42d9-b140-960ad92a954a","Type":"ContainerStarted","Data":"25505c76838304350d610d34a16adbb1b4f5bab664debad37104cdad65e2a583"}
Oct 30 13:10:05.835493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.835313    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2" event={"ID":"812684fe-ef9c-42d9-b140-960ad92a954a","Type":"ContainerStarted","Data":"4b8a9c96f60341ea747fd5fd611cfdf3dd0f89de83aa4356e86354ae8300f07a"}
Oct 30 13:10:05.898037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:05.897979    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2" podStartSLOduration=1.8979610770000002 podStartE2EDuration="1.897961077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:05.89613405 +0000 UTC m=+2270.371853660" watchObservedRunningTime="2026-10-30 13:10:05.897961077 +0000 UTC m=+2270.373680687"
Oct 30 13:10:06.852858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:06.852814    4993 generic.go:356] "Generic (PLEG): container finished" podID="812684fe-ef9c-42d9-b140-960ad92a954a" containerID="25505c76838304350d610d34a16adbb1b4f5bab664debad37104cdad65e2a583" exitCode=0
Oct 30 13:10:06.853416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:06.852895    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2" event={"ID":"812684fe-ef9c-42d9-b140-960ad92a954a","Type":"ContainerDied","Data":"25505c76838304350d610d34a16adbb1b4f5bab664debad37104cdad65e2a583"}
Oct 30 13:10:07.155706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.155623    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"]
Oct 30 13:10:07.159615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.159593    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.180686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.180650    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"]
Oct 30 13:10:07.274113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.274066    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkz88\" (UniqueName: \"kubernetes.io/projected/0565be25-44d5-4648-b9af-81985b0cd8bc-kube-api-access-rkz88\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.274328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.274129    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.274328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.274280    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.375328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.375293    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkz88\" (UniqueName: \"kubernetes.io/projected/0565be25-44d5-4648-b9af-81985b0cd8bc-kube-api-access-rkz88\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.375328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.375341    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.375628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.375387    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.376056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.376032    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.378125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.378087    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.401284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.401242    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkz88\" (UniqueName: \"kubernetes.io/projected/0565be25-44d5-4648-b9af-81985b0cd8bc-kube-api-access-rkz88\") pod \"maas-api-revoke-sub-fad1f1aa-qm78l\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.474119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.474032    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:07.803306 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.803088    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"]
Oct 30 13:10:07.869683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:07.869633    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l" event={"ID":"0565be25-44d5-4648-b9af-81985b0cd8bc","Type":"ContainerStarted","Data":"016e7b366952c5ccfec31890a7ec3dbfdb6f16de75de940451eca2d8d3804e64"}
Oct 30 13:10:08.019045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.019016    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:08.084682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.084578    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-xks-ca\") pod \"812684fe-ef9c-42d9-b140-960ad92a954a\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") "
Oct 30 13:10:08.084682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.084662    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812684fe-ef9c-42d9-b140-960ad92a954a-kube-api-access-q2h5z\" (UniqueName: \"kubernetes.io/projected/812684fe-ef9c-42d9-b140-960ad92a954a-kube-api-access-q2h5z\") pod \"812684fe-ef9c-42d9-b140-960ad92a954a\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") "
Oct 30 13:10:08.084927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.084801    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-openshift-ca\") pod \"812684fe-ef9c-42d9-b140-960ad92a954a\" (UID: \"812684fe-ef9c-42d9-b140-960ad92a954a\") "
Oct 30 13:10:08.085200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.085171    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-openshift-ca" pod "812684fe-ef9c-42d9-b140-960ad92a954a" (UID: "812684fe-ef9c-42d9-b140-960ad92a954a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:08.085467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.085445    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:08.087548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.087516    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-xks-ca" pod "812684fe-ef9c-42d9-b140-960ad92a954a" (UID: "812684fe-ef9c-42d9-b140-960ad92a954a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:08.087548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.087524    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812684fe-ef9c-42d9-b140-960ad92a954a-kube-api-access-q2h5z" pod "812684fe-ef9c-42d9-b140-960ad92a954a" (UID: "812684fe-ef9c-42d9-b140-960ad92a954a"). InnerVolumeSpecName "kube-api-access-q2h5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:08.186733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.186693    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/812684fe-ef9c-42d9-b140-960ad92a954a-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:08.186733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.186733    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2h5z\" (UniqueName: \"kubernetes.io/projected/812684fe-ef9c-42d9-b140-960ad92a954a-kube-api-access-q2h5z\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:08.582290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.581549    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"]
Oct 30 13:10:08.583166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.583140    4993 cpu_manager.go:414] "containerMap: removing container" podUID="812684fe-ef9c-42d9-b140-960ad92a954a" containerName="revoke-keys"
Oct 30 13:10:08.583307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.583169    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812684fe-ef9c-42d9-b140-960ad92a954a" containerName="revoke-keys"
Oct 30 13:10:08.583469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.583450    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="812684fe-ef9c-42d9-b140-960ad92a954a" containerName="revoke-keys"
Oct 30 13:10:08.586533 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:10:08.586489    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:10:08.587657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.587634    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.609199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.609133    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"]
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710323    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710437    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710467    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710542    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwb9g\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-kube-api-access-xwb9g\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710582    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710664    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710730    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710755    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6349107-300d-4744-8712-9a0712a45e7e-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.716286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.710821    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6349107-300d-4744-8712-9a0712a45e7e-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814098    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814171    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814191    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814269    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwb9g\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-kube-api-access-xwb9g\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814292    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814345    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814386    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814400    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6349107-300d-4744-8712-9a0712a45e7e-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814440    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6349107-300d-4744-8712-9a0712a45e7e-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.814663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.814585    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.816390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.815111    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.816390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.815241    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.816390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.815365    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.816390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.816255    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6349107-300d-4744-8712-9a0712a45e7e-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.817658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.817630    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6349107-300d-4744-8712-9a0712a45e7e-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.818461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.818438    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.838382 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:10:08.838343    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0565be25_44d5_4648_b9af_81985b0cd8bc.slice/crio-conmon-17eacce0a4680dda7fcaa1a97806650cf471abe8548f0c7eca392d1eb4a4b77a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0565be25_44d5_4648_b9af_81985b0cd8bc.slice/crio-17eacce0a4680dda7fcaa1a97806650cf471abe8548f0c7eca392d1eb4a4b77a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:10:08.840009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.839972    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.842449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.842424    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwb9g\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-kube-api-access-xwb9g\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:08.881430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.881394    4993 generic.go:356] "Generic (PLEG): container finished" podID="0565be25-44d5-4648-b9af-81985b0cd8bc" containerID="17eacce0a4680dda7fcaa1a97806650cf471abe8548f0c7eca392d1eb4a4b77a" exitCode=0
Oct 30 13:10:08.881636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.881478    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l" event={"ID":"0565be25-44d5-4648-b9af-81985b0cd8bc","Type":"ContainerDied","Data":"17eacce0a4680dda7fcaa1a97806650cf471abe8548f0c7eca392d1eb4a4b77a"}
Oct 30 13:10:08.883064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.883035    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"
Oct 30 13:10:08.883189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.883074    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2" event={"ID":"812684fe-ef9c-42d9-b140-960ad92a954a","Type":"ContainerDied","Data":"4b8a9c96f60341ea747fd5fd611cfdf3dd0f89de83aa4356e86354ae8300f07a"}
Oct 30 13:10:08.883189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.883127    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8a9c96f60341ea747fd5fd611cfdf3dd0f89de83aa4356e86354ae8300f07a"
Oct 30 13:10:08.926532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:08.926492    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:09.266250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:09.261717    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"]
Oct 30 13:10:09.268486 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:09.268428    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6349107_300d_4744_8712_9a0712a45e7e.slice/crio-40a066d6159e350ed8e870375e1cae9bd05c89f4ae8ac47d8c4d600afa44fdf3 WatchSource:0}: Error finding container 40a066d6159e350ed8e870375e1cae9bd05c89f4ae8ac47d8c4d600afa44fdf3: Status 404 returned error can't find the container with id 40a066d6159e350ed8e870375e1cae9bd05c89f4ae8ac47d8c4d600afa44fdf3
Oct 30 13:10:09.271616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:09.271576    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:09.271738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:09.271655    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:09.271738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:09.271685    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:09.893730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:09.893685    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" event={"ID":"e6349107-300d-4744-8712-9a0712a45e7e","Type":"ContainerStarted","Data":"ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa"}
Oct 30 13:10:09.893923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:09.893754    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" event={"ID":"e6349107-300d-4744-8712-9a0712a45e7e","Type":"ContainerStarted","Data":"40a066d6159e350ed8e870375e1cae9bd05c89f4ae8ac47d8c4d600afa44fdf3"}
Oct 30 13:10:09.926936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:09.926895    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:09.947605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:09.947535    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podStartSLOduration=1.9475141470000001 podStartE2EDuration="1.947514147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:09.945099524 +0000 UTC m=+2274.420819135" watchObservedRunningTime="2026-10-30 13:10:09.947514147 +0000 UTC m=+2274.423233757"
Oct 30 13:10:10.071550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.071510    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:10.134841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.134805    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-xks-ca\") pod \"0565be25-44d5-4648-b9af-81985b0cd8bc\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") "
Oct 30 13:10:10.135050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.134883    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-openshift-ca\") pod \"0565be25-44d5-4648-b9af-81985b0cd8bc\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") "
Oct 30 13:10:10.135118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.135052    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0565be25-44d5-4648-b9af-81985b0cd8bc-kube-api-access-rkz88\" (UniqueName: \"kubernetes.io/projected/0565be25-44d5-4648-b9af-81985b0cd8bc-kube-api-access-rkz88\") pod \"0565be25-44d5-4648-b9af-81985b0cd8bc\" (UID: \"0565be25-44d5-4648-b9af-81985b0cd8bc\") "
Oct 30 13:10:10.135373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.135341    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-openshift-ca" pod "0565be25-44d5-4648-b9af-81985b0cd8bc" (UID: "0565be25-44d5-4648-b9af-81985b0cd8bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:10.140231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.135789    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:10.140231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.138393    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0565be25-44d5-4648-b9af-81985b0cd8bc-kube-api-access-rkz88" pod "0565be25-44d5-4648-b9af-81985b0cd8bc" (UID: "0565be25-44d5-4648-b9af-81985b0cd8bc"). InnerVolumeSpecName "kube-api-access-rkz88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:10.140231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.139078    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-xks-ca" pod "0565be25-44d5-4648-b9af-81985b0cd8bc" (UID: "0565be25-44d5-4648-b9af-81985b0cd8bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:10.237440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.237395    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkz88\" (UniqueName: \"kubernetes.io/projected/0565be25-44d5-4648-b9af-81985b0cd8bc-kube-api-access-rkz88\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:10.237629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.237456    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0565be25-44d5-4648-b9af-81985b0cd8bc-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:10.914231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.912742    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"
Oct 30 13:10:10.914573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.914539    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l" event={"ID":"0565be25-44d5-4648-b9af-81985b0cd8bc","Type":"ContainerDied","Data":"016e7b366952c5ccfec31890a7ec3dbfdb6f16de75de940451eca2d8d3804e64"}
Oct 30 13:10:10.914955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.914936    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016e7b366952c5ccfec31890a7ec3dbfdb6f16de75de940451eca2d8d3804e64"
Oct 30 13:10:10.928833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.928488    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:10:10.928833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:10.928565    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 13:10:11.709457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:11.708828    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"]
Oct 30 13:10:11.709457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:11.708879    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9c553-zk98n"]
Oct 30 13:10:11.927350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:11.927303    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:10:11.927685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:11.927645    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 13:10:12.792509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.792471    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"]
Oct 30 13:10:12.794674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.794640    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0565be25-44d5-4648-b9af-81985b0cd8bc" containerName="revoke-keys"
Oct 30 13:10:12.794674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.794673    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0565be25-44d5-4648-b9af-81985b0cd8bc" containerName="revoke-keys"
Oct 30 13:10:12.795191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.795166    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0565be25-44d5-4648-b9af-81985b0cd8bc" containerName="revoke-keys"
Oct 30 13:10:12.808765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.808738    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:12.816676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.816638    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"]
Oct 30 13:10:12.826060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.826031    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:12.826521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.826502    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:10:12.826623 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.826562    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:12.839299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.839257    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:12.927045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.927005    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:10:12.927256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.927072    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 13:10:12.947286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.947199    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:12.947488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.947395    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsvv\" (UniqueName: \"kubernetes.io/projected/38ec7be5-ae82-45c1-87c2-b5ee6726d692-kube-api-access-xfsvv\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:12.947488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:12.947483    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:13.048447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.048354    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsvv\" (UniqueName: \"kubernetes.io/projected/38ec7be5-ae82-45c1-87c2-b5ee6726d692-kube-api-access-xfsvv\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:13.048447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.048434    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:13.048692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.048567    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:13.049291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.049258    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:13.050954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.050927    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:13.097502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.097468    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsvv\" (UniqueName: \"kubernetes.io/projected/38ec7be5-ae82-45c1-87c2-b5ee6726d692-kube-api-access-xfsvv\") pod \"maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:13.125015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.124974    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:13.510918 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:13.510884    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ec7be5_ae82_45c1_87c2_b5ee6726d692.slice/crio-66a3beff469c81e2f5982f737dec0ba46bc470970e5f054ec1eb9342af237108 WatchSource:0}: Error finding container 66a3beff469c81e2f5982f737dec0ba46bc470970e5f054ec1eb9342af237108: Status 404 returned error can't find the container with id 66a3beff469c81e2f5982f737dec0ba46bc470970e5f054ec1eb9342af237108
Oct 30 13:10:13.511105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.510924    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"]
Oct 30 13:10:13.699315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.699282    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18989a21-9740-4fbd-8440-24d8ab33615c" path="/var/lib/kubelet/pods/18989a21-9740-4fbd-8440-24d8ab33615c/volumes"
Oct 30 13:10:13.927859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.927822    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:10:13.928279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.927882    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 13:10:13.938808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.938761    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v" event={"ID":"38ec7be5-ae82-45c1-87c2-b5ee6726d692","Type":"ContainerStarted","Data":"d1c6acc20cfae7c03264e80b421bc41b9d32f77b8c26989398f3488d3652697f"}
Oct 30 13:10:13.938968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.938825    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v" event={"ID":"38ec7be5-ae82-45c1-87c2-b5ee6726d692","Type":"ContainerStarted","Data":"66a3beff469c81e2f5982f737dec0ba46bc470970e5f054ec1eb9342af237108"}
Oct 30 13:10:13.992260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:13.992174    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v" podStartSLOduration=1.992154178 podStartE2EDuration="1.992154178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:13.990622972 +0000 UTC m=+2278.466342595" watchObservedRunningTime="2026-10-30 13:10:13.992154178 +0000 UTC m=+2278.467873789"
Oct 30 13:10:14.927303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:14.927185    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:10:14.927303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:14.927255    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 13:10:14.948179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:14.948146    4993 generic.go:356] "Generic (PLEG): container finished" podID="38ec7be5-ae82-45c1-87c2-b5ee6726d692" containerID="d1c6acc20cfae7c03264e80b421bc41b9d32f77b8c26989398f3488d3652697f" exitCode=0
Oct 30 13:10:14.948577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:14.948255    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v" event={"ID":"38ec7be5-ae82-45c1-87c2-b5ee6726d692","Type":"ContainerDied","Data":"d1c6acc20cfae7c03264e80b421bc41b9d32f77b8c26989398f3488d3652697f"}
Oct 30 13:10:15.927715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:15.927672    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:10:15.927882 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:15.927738    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 13:10:16.212926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.212890    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:16.272821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.272775    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-xks-ca\") pod \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") "
Oct 30 13:10:16.273037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.272868    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ec7be5-ae82-45c1-87c2-b5ee6726d692-kube-api-access-xfsvv\" (UniqueName: \"kubernetes.io/projected/38ec7be5-ae82-45c1-87c2-b5ee6726d692-kube-api-access-xfsvv\") pod \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") "
Oct 30 13:10:16.273037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.272986    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-openshift-ca\") pod \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\" (UID: \"38ec7be5-ae82-45c1-87c2-b5ee6726d692\") "
Oct 30 13:10:16.273900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.273852    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-openshift-ca" pod "38ec7be5-ae82-45c1-87c2-b5ee6726d692" (UID: "38ec7be5-ae82-45c1-87c2-b5ee6726d692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:16.276677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.276642    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-xks-ca" pod "38ec7be5-ae82-45c1-87c2-b5ee6726d692" (UID: "38ec7be5-ae82-45c1-87c2-b5ee6726d692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:16.277517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.277490    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ec7be5-ae82-45c1-87c2-b5ee6726d692-kube-api-access-xfsvv" pod "38ec7be5-ae82-45c1-87c2-b5ee6726d692" (UID: "38ec7be5-ae82-45c1-87c2-b5ee6726d692"). InnerVolumeSpecName "kube-api-access-xfsvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:16.387760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.376106    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:16.387760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.376150    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfsvv\" (UniqueName: \"kubernetes.io/projected/38ec7be5-ae82-45c1-87c2-b5ee6726d692-kube-api-access-xfsvv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:16.387760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.376180    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec7be5-ae82-45c1-87c2-b5ee6726d692-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:16.943318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.941378    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:10:16.943318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:16.941448    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 13:10:17.073262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:17.071399    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"
Oct 30 13:10:17.085229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:17.039188    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v" event={"ID":"38ec7be5-ae82-45c1-87c2-b5ee6726d692","Type":"ContainerDied","Data":"66a3beff469c81e2f5982f737dec0ba46bc470970e5f054ec1eb9342af237108"}
Oct 30 13:10:17.085229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:17.078018    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a3beff469c81e2f5982f737dec0ba46bc470970e5f054ec1eb9342af237108"
Oct 30 13:10:17.985232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:17.984822    4993 scope.go:122] "RemoveContainer" containerID="c05a81a9d44eac3789db9854210d3b34003e2dd97760a448ccc62eb815c58867"
Oct 30 13:10:18.598588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:18.598553    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:18.601840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:18.601813    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:18.603494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:18.603470    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:10:19.179598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:19.179538    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:19.258410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:19.258349    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:19.428428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:19.428385    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:10:19.428706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:19.428663    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:21.035437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.035394    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"]
Oct 30 13:10:21.035993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.035979    4993 cpu_manager.go:414] "containerMap: removing container" podUID="38ec7be5-ae82-45c1-87c2-b5ee6726d692" containerName="revoke-keys"
Oct 30 13:10:21.036040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.035995    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ec7be5-ae82-45c1-87c2-b5ee6726d692" containerName="revoke-keys"
Oct 30 13:10:21.036115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.036106    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ec7be5-ae82-45c1-87c2-b5ee6726d692" containerName="revoke-keys"
Oct 30 13:10:21.041060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.041040    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.076540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.076505    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:21.076891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.076872    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:21.085335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.077890    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:21.085335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.077953    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:10:21.085335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.082340    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"]
Oct 30 13:10:21.239680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.239636    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbjw\" (UniqueName: \"kubernetes.io/projected/4169f47f-be91-488c-a607-65a6253e3187-kube-api-access-tjbjw\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.239982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.239950    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4169f47f-be91-488c-a607-65a6253e3187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.240075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.240051    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4169f47f-be91-488c-a607-65a6253e3187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.341357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.341195    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4169f47f-be91-488c-a607-65a6253e3187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.341357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.341336    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4169f47f-be91-488c-a607-65a6253e3187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.341595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.341480    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbjw\" (UniqueName: \"kubernetes.io/projected/4169f47f-be91-488c-a607-65a6253e3187-kube-api-access-tjbjw\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.342075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.342039    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4169f47f-be91-488c-a607-65a6253e3187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.343891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.343863    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4169f47f-be91-488c-a607-65a6253e3187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.367305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.367265    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbjw\" (UniqueName: \"kubernetes.io/projected/4169f47f-be91-488c-a607-65a6253e3187-kube-api-access-tjbjw\") pod \"maas-api-revoke-sub-2a59fa51-cc4ch\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.380136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.380101    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:21.699387 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:21.699354    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4169f47f_be91_488c_a607_65a6253e3187.slice/crio-a49977baf78c3060790abb8ba720e0721dec17f46579496094db25e49bb3c4af WatchSource:0}: Error finding container a49977baf78c3060790abb8ba720e0721dec17f46579496094db25e49bb3c4af: Status 404 returned error can't find the container with id a49977baf78c3060790abb8ba720e0721dec17f46579496094db25e49bb3c4af
Oct 30 13:10:21.701632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:21.701568    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"]
Oct 30 13:10:22.232639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:22.232590    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch" event={"ID":"4169f47f-be91-488c-a607-65a6253e3187","Type":"ContainerStarted","Data":"1fe58ce27797e5890905e98cd761e658fb572abf3c231faa1619d341926cbfd4"}
Oct 30 13:10:22.233117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:22.232693    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch" event={"ID":"4169f47f-be91-488c-a607-65a6253e3187","Type":"ContainerStarted","Data":"a49977baf78c3060790abb8ba720e0721dec17f46579496094db25e49bb3c4af"}
Oct 30 13:10:22.288392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:22.288332    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch" podStartSLOduration=1.2883181160000001 podStartE2EDuration="1.288318116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:22.286585511 +0000 UTC m=+2286.762305122" watchObservedRunningTime="2026-10-30 13:10:22.288318116 +0000 UTC m=+2286.764037726"
Oct 30 13:10:23.242548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:23.242513    4993 generic.go:356] "Generic (PLEG): container finished" podID="4169f47f-be91-488c-a607-65a6253e3187" containerID="1fe58ce27797e5890905e98cd761e658fb572abf3c231faa1619d341926cbfd4" exitCode=0
Oct 30 13:10:23.242992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:23.242616    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch" event={"ID":"4169f47f-be91-488c-a607-65a6253e3187","Type":"ContainerDied","Data":"1fe58ce27797e5890905e98cd761e658fb572abf3c231faa1619d341926cbfd4"}
Oct 30 13:10:24.464700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.464665    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:24.504902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.504818    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4169f47f-be91-488c-a607-65a6253e3187-kube-api-access-tjbjw\" (UniqueName: \"kubernetes.io/projected/4169f47f-be91-488c-a607-65a6253e3187-kube-api-access-tjbjw\") pod \"4169f47f-be91-488c-a607-65a6253e3187\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") "
Oct 30 13:10:24.504902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.504878    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4169f47f-be91-488c-a607-65a6253e3187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4169f47f-be91-488c-a607-65a6253e3187-maas-api-openshift-ca\") pod \"4169f47f-be91-488c-a607-65a6253e3187\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") "
Oct 30 13:10:24.505147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.504911    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4169f47f-be91-488c-a607-65a6253e3187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4169f47f-be91-488c-a607-65a6253e3187-maas-api-xks-ca\") pod \"4169f47f-be91-488c-a607-65a6253e3187\" (UID: \"4169f47f-be91-488c-a607-65a6253e3187\") "
Oct 30 13:10:24.505258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.505233    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4169f47f-be91-488c-a607-65a6253e3187-maas-api-openshift-ca" pod "4169f47f-be91-488c-a607-65a6253e3187" (UID: "4169f47f-be91-488c-a607-65a6253e3187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:24.505592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.505575    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4169f47f-be91-488c-a607-65a6253e3187-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:24.506998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.506961    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4169f47f-be91-488c-a607-65a6253e3187-maas-api-xks-ca" pod "4169f47f-be91-488c-a607-65a6253e3187" (UID: "4169f47f-be91-488c-a607-65a6253e3187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:24.506998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.506974    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4169f47f-be91-488c-a607-65a6253e3187-kube-api-access-tjbjw" pod "4169f47f-be91-488c-a607-65a6253e3187" (UID: "4169f47f-be91-488c-a607-65a6253e3187"). InnerVolumeSpecName "kube-api-access-tjbjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:24.607524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.607486    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4169f47f-be91-488c-a607-65a6253e3187-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:24.607524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.607519    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjbjw\" (UniqueName: \"kubernetes.io/projected/4169f47f-be91-488c-a607-65a6253e3187-kube-api-access-tjbjw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:24.809984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.809895    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"]
Oct 30 13:10:24.810160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.810114    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy" containerID="cri-o://6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff" gracePeriod=30
Oct 30 13:10:24.984993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.984952    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"]
Oct 30 13:10:24.985883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.985859    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4169f47f-be91-488c-a607-65a6253e3187" containerName="revoke-keys"
Oct 30 13:10:24.985979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.985885    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4169f47f-be91-488c-a607-65a6253e3187" containerName="revoke-keys"
Oct 30 13:10:24.986082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.986069    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4169f47f-be91-488c-a607-65a6253e3187" containerName="revoke-keys"
Oct 30 13:10:24.990512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:24.990484    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.008532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.008491    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"]
Oct 30 13:10:25.114071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.113970    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwqq\" (UniqueName: \"kubernetes.io/projected/fb31659b-ab4b-44c7-b7ee-685f3115c220-kube-api-access-cjwqq\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.114251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.114086    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.114378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.114358    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.215621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.215584    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.215772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.215689    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwqq\" (UniqueName: \"kubernetes.io/projected/fb31659b-ab4b-44c7-b7ee-685f3115c220-kube-api-access-cjwqq\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.215772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.215759    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.216440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.216415    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.218359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.218332    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.243539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.243507    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwqq\" (UniqueName: \"kubernetes.io/projected/fb31659b-ab4b-44c7-b7ee-685f3115c220-kube-api-access-cjwqq\") pod \"maas-api-revoke-sub-5bbec39d-n6wlp\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.263646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.263617    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"
Oct 30 13:10:25.263803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.263611    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch" event={"ID":"4169f47f-be91-488c-a607-65a6253e3187","Type":"ContainerDied","Data":"a49977baf78c3060790abb8ba720e0721dec17f46579496094db25e49bb3c4af"}
Oct 30 13:10:25.263803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.263770    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a49977baf78c3060790abb8ba720e0721dec17f46579496094db25e49bb3c4af"
Oct 30 13:10:25.309891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.309853    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:25.618822 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:25.618780    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb31659b_ab4b_44c7_b7ee_685f3115c220.slice/crio-71aabdcfc37533816311c70350a26271ec4a55cf2a45f772cb105b68d0544823 WatchSource:0}: Error finding container 71aabdcfc37533816311c70350a26271ec4a55cf2a45f772cb105b68d0544823: Status 404 returned error can't find the container with id 71aabdcfc37533816311c70350a26271ec4a55cf2a45f772cb105b68d0544823
Oct 30 13:10:25.621822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:25.621783    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"]
Oct 30 13:10:26.278548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:26.278508    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp" event={"ID":"fb31659b-ab4b-44c7-b7ee-685f3115c220","Type":"ContainerStarted","Data":"ab2f97cd490f77924297c1c4d27454c219206320473f7c413f0f25b6bea1da60"}
Oct 30 13:10:26.278752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:26.278576    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp" event={"ID":"fb31659b-ab4b-44c7-b7ee-685f3115c220","Type":"ContainerStarted","Data":"71aabdcfc37533816311c70350a26271ec4a55cf2a45f772cb105b68d0544823"}
Oct 30 13:10:26.325061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:26.324997    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp" podStartSLOduration=2.32498095 podStartE2EDuration="2.32498095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:26.322948758 +0000 UTC m=+2290.798668372" watchObservedRunningTime="2026-10-30 13:10:26.32498095 +0000 UTC m=+2290.800700551"
Oct 30 13:10:27.291194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:27.291155    4993 generic.go:356] "Generic (PLEG): container finished" podID="fb31659b-ab4b-44c7-b7ee-685f3115c220" containerID="ab2f97cd490f77924297c1c4d27454c219206320473f7c413f0f25b6bea1da60" exitCode=0
Oct 30 13:10:27.291726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:27.291237    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp" event={"ID":"fb31659b-ab4b-44c7-b7ee-685f3115c220","Type":"ContainerDied","Data":"ab2f97cd490f77924297c1c4d27454c219206320473f7c413f0f25b6bea1da60"}
Oct 30 13:10:28.483134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.483103    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:28.566272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.566153    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-xks-ca\") pod \"fb31659b-ab4b-44c7-b7ee-685f3115c220\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") "
Oct 30 13:10:28.566457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.566335    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb31659b-ab4b-44c7-b7ee-685f3115c220-kube-api-access-cjwqq\" (UniqueName: \"kubernetes.io/projected/fb31659b-ab4b-44c7-b7ee-685f3115c220-kube-api-access-cjwqq\") pod \"fb31659b-ab4b-44c7-b7ee-685f3115c220\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") "
Oct 30 13:10:28.566534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.566520    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-openshift-ca\") pod \"fb31659b-ab4b-44c7-b7ee-685f3115c220\" (UID: \"fb31659b-ab4b-44c7-b7ee-685f3115c220\") "
Oct 30 13:10:28.567020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.566986    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-openshift-ca" pod "fb31659b-ab4b-44c7-b7ee-685f3115c220" (UID: "fb31659b-ab4b-44c7-b7ee-685f3115c220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:28.568669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.568641    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-xks-ca" pod "fb31659b-ab4b-44c7-b7ee-685f3115c220" (UID: "fb31659b-ab4b-44c7-b7ee-685f3115c220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:28.568939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.568921    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb31659b-ab4b-44c7-b7ee-685f3115c220-kube-api-access-cjwqq" pod "fb31659b-ab4b-44c7-b7ee-685f3115c220" (UID: "fb31659b-ab4b-44c7-b7ee-685f3115c220"). InnerVolumeSpecName "kube-api-access-cjwqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:28.668475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.668432    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjwqq\" (UniqueName: \"kubernetes.io/projected/fb31659b-ab4b-44c7-b7ee-685f3115c220-kube-api-access-cjwqq\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:28.668475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.668474    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:28.668724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:28.668494    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb31659b-ab4b-44c7-b7ee-685f3115c220-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:29.347362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:29.342493    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp" event={"ID":"fb31659b-ab4b-44c7-b7ee-685f3115c220","Type":"ContainerDied","Data":"71aabdcfc37533816311c70350a26271ec4a55cf2a45f772cb105b68d0544823"}
Oct 30 13:10:29.347362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:29.342592    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71aabdcfc37533816311c70350a26271ec4a55cf2a45f772cb105b68d0544823"
Oct 30 13:10:29.347362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:29.342657    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"
Oct 30 13:10:29.790470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:29.790425    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"]
Oct 30 13:10:29.802955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:29.802919    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d84f47-nq27v"]
Oct 30 13:10:30.102011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.101983    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 13:10:30.248845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.248809    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-credential-socket\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.248845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.248859    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-certs\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.249123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.248883    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-data\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.249123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.248906    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-socket\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.249123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.248941    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d5d695-584f-40a5-92a0-eb12f0b5b959-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d5d695-584f-40a5-92a0-eb12f0b5b959-istiod-ca-cert\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.249123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.248995    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-podinfo\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.249123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249056    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-envoy\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.249123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249123    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-token\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-token\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.249472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249131    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-credential-socket" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:30.249472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249263    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-kube-api-access-vv4kx\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-kube-api-access-vv4kx\") pod \"02d5d695-584f-40a5-92a0-eb12f0b5b959\" (UID: \"02d5d695-584f-40a5-92a0-eb12f0b5b959\") "
Oct 30 13:10:30.249472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249314    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-socket" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:30.249472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249327    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-certs" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:30.249472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249330    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-data" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:30.249731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249467    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d5d695-584f-40a5-92a0-eb12f0b5b959-istiod-ca-cert" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:30.249965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249946    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.250048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249968    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.250048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249982    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.250048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.249993    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.250048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.250005    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d5d695-584f-40a5-92a0-eb12f0b5b959-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.251430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.251392    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-podinfo" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:30.251551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.251489    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-token" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:30.251853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.251836    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-envoy" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:30.251920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.251863    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-kube-api-access-vv4kx" pod "02d5d695-584f-40a5-92a0-eb12f0b5b959" (UID: "02d5d695-584f-40a5-92a0-eb12f0b5b959"). InnerVolumeSpecName "kube-api-access-vv4kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:30.351656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.351557    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.351656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.351594    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.351656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.351607    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.351656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.351619    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv4kx\" (UniqueName: \"kubernetes.io/projected/02d5d695-584f-40a5-92a0-eb12f0b5b959-kube-api-access-vv4kx\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:30.354739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.354706    4993 generic.go:356] "Generic (PLEG): container finished" podID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerID="6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff" exitCode=0
Oct 30 13:10:30.354938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.354773    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"
Oct 30 13:10:30.354938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.354837    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" event={"ID":"02d5d695-584f-40a5-92a0-eb12f0b5b959","Type":"ContainerDied","Data":"6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff"}
Oct 30 13:10:30.354938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.354908    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h" event={"ID":"02d5d695-584f-40a5-92a0-eb12f0b5b959","Type":"ContainerDied","Data":"94027c6119737f8d14409010dbc8db9e9c49aab083d1d81d547f013a02a10f73"}
Oct 30 13:10:30.354938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.354931    4993 scope.go:122] "RemoveContainer" containerID="6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff"
Oct 30 13:10:30.367768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.367743    4993 scope.go:122] "RemoveContainer" containerID="778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee"
Oct 30 13:10:30.379054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.379032    4993 scope.go:122] "RemoveContainer" containerID="6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff"
Oct 30 13:10:30.379483 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:10:30.379450    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff\": container with ID starting with 6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff not found: ID does not exist" containerID="6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff"
Oct 30 13:10:30.379590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.379499    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff"} err="failed to get container status \"6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff\": rpc error: code = NotFound desc = could not find container \"6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff\": container with ID starting with 6f0b45e9189fc86c41ce0e18b68654f3fa2b719d76c39c920130b0481db730ff not found: ID does not exist"
Oct 30 13:10:30.379590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.379527    4993 scope.go:122] "RemoveContainer" containerID="778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee"
Oct 30 13:10:30.379860 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:10:30.379832    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee\": container with ID starting with 778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee not found: ID does not exist" containerID="778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee"
Oct 30 13:10:30.379909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.379867    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee"} err="failed to get container status \"778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee\": rpc error: code = NotFound desc = could not find container \"778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee\": container with ID starting with 778e1ec84551fb809d0ce27c7917ff1a2528a78dd2a8047a62712e6d54abd4ee not found: ID does not exist"
Oct 30 13:10:30.452538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.452499    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"]
Oct 30 13:10:30.465843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:30.465806    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-db46h"]
Oct 30 13:10:31.692690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:31.692656    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" path="/var/lib/kubelet/pods/02d5d695-584f-40a5-92a0-eb12f0b5b959/volumes"
Oct 30 13:10:31.693347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:31.693328    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ec7be5-ae82-45c1-87c2-b5ee6726d692" path="/var/lib/kubelet/pods/38ec7be5-ae82-45c1-87c2-b5ee6726d692/volumes"
Oct 30 13:10:32.938347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.938312    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"]
Oct 30 13:10:32.938924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.938909    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fb31659b-ab4b-44c7-b7ee-685f3115c220" containerName="revoke-keys"
Oct 30 13:10:32.938968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.938925    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb31659b-ab4b-44c7-b7ee-685f3115c220" containerName="revoke-keys"
Oct 30 13:10:32.938968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.938941    4993 cpu_manager.go:414] "containerMap: removing container" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy"
Oct 30 13:10:32.938968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.938946    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy"
Oct 30 13:10:32.938968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.938963    4993 cpu_manager.go:414] "containerMap: removing container" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy"
Oct 30 13:10:32.939093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.938969    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy"
Oct 30 13:10:32.939093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.939050    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb31659b-ab4b-44c7-b7ee-685f3115c220" containerName="revoke-keys"
Oct 30 13:10:32.939093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.939060    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy"
Oct 30 13:10:32.939093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.939068    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d5d695-584f-40a5-92a0-eb12f0b5b959" containerName="istio-proxy"
Oct 30 13:10:32.944004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.943976    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:32.960629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.960598    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:32.963435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.963396    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"]
Oct 30 13:10:32.971972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.971939    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:32.972185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.972086    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:10:32.972185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.972176    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:32.985764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.985719    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:32.985971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.985887    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j595b\" (UniqueName: \"kubernetes.io/projected/f1a146a5-cd63-4426-a5ad-63a7018c3261-kube-api-access-j595b\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:32.985971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:32.985928    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:33.087717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:33.087670    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j595b\" (UniqueName: \"kubernetes.io/projected/f1a146a5-cd63-4426-a5ad-63a7018c3261-kube-api-access-j595b\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:33.087910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:33.087737    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:33.087957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:33.087941    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:33.088710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:33.088685    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:33.090403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:33.090384    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:33.114185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:33.114143    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j595b\" (UniqueName: \"kubernetes.io/projected/f1a146a5-cd63-4426-a5ad-63a7018c3261-kube-api-access-j595b\") pod \"maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:33.258355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:33.258314    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:33.581321 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:33.581278    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a146a5_cd63_4426_a5ad_63a7018c3261.slice/crio-09f449ab076398bd874f3da44870a7a2ef0b31410aec1eaf0fc8a73cd701d7ca WatchSource:0}: Error finding container 09f449ab076398bd874f3da44870a7a2ef0b31410aec1eaf0fc8a73cd701d7ca: Status 404 returned error can't find the container with id 09f449ab076398bd874f3da44870a7a2ef0b31410aec1eaf0fc8a73cd701d7ca
Oct 30 13:10:33.583099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:33.583062    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"]
Oct 30 13:10:34.394527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:34.394473    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt" event={"ID":"f1a146a5-cd63-4426-a5ad-63a7018c3261","Type":"ContainerStarted","Data":"d09842d03b9928a6ae1c6b2306951171d49b619b5ea78cf85c3833bfa5a292b2"}
Oct 30 13:10:34.395034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:34.394548    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt" event={"ID":"f1a146a5-cd63-4426-a5ad-63a7018c3261","Type":"ContainerStarted","Data":"09f449ab076398bd874f3da44870a7a2ef0b31410aec1eaf0fc8a73cd701d7ca"}
Oct 30 13:10:34.439250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:34.439172    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt" podStartSLOduration=2.439158112 podStartE2EDuration="2.439158112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:34.438683156 +0000 UTC m=+2298.914402766" watchObservedRunningTime="2026-10-30 13:10:34.439158112 +0000 UTC m=+2298.914877722"
Oct 30 13:10:35.405747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:35.405712    4993 generic.go:356] "Generic (PLEG): container finished" podID="f1a146a5-cd63-4426-a5ad-63a7018c3261" containerID="d09842d03b9928a6ae1c6b2306951171d49b619b5ea78cf85c3833bfa5a292b2" exitCode=0
Oct 30 13:10:35.406274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:35.405810    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt" event={"ID":"f1a146a5-cd63-4426-a5ad-63a7018c3261","Type":"ContainerDied","Data":"d09842d03b9928a6ae1c6b2306951171d49b619b5ea78cf85c3833bfa5a292b2"}
Oct 30 13:10:36.616889 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.616851    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:36.635315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.635276    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a146a5-cd63-4426-a5ad-63a7018c3261-kube-api-access-j595b\" (UniqueName: \"kubernetes.io/projected/f1a146a5-cd63-4426-a5ad-63a7018c3261-kube-api-access-j595b\") pod \"f1a146a5-cd63-4426-a5ad-63a7018c3261\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") "
Oct 30 13:10:36.635505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.635442    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-openshift-ca\") pod \"f1a146a5-cd63-4426-a5ad-63a7018c3261\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") "
Oct 30 13:10:36.635662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.635639    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-xks-ca\") pod \"f1a146a5-cd63-4426-a5ad-63a7018c3261\" (UID: \"f1a146a5-cd63-4426-a5ad-63a7018c3261\") "
Oct 30 13:10:36.635825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.635798    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-openshift-ca" pod "f1a146a5-cd63-4426-a5ad-63a7018c3261" (UID: "f1a146a5-cd63-4426-a5ad-63a7018c3261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:36.636177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.636154    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:36.637776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.637752    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-xks-ca" pod "f1a146a5-cd63-4426-a5ad-63a7018c3261" (UID: "f1a146a5-cd63-4426-a5ad-63a7018c3261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:36.637873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.637778    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a146a5-cd63-4426-a5ad-63a7018c3261-kube-api-access-j595b" pod "f1a146a5-cd63-4426-a5ad-63a7018c3261" (UID: "f1a146a5-cd63-4426-a5ad-63a7018c3261"). InnerVolumeSpecName "kube-api-access-j595b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:36.737436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.737398    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j595b\" (UniqueName: \"kubernetes.io/projected/f1a146a5-cd63-4426-a5ad-63a7018c3261-kube-api-access-j595b\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:36.737436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:36.737435    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a146a5-cd63-4426-a5ad-63a7018c3261-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:37.427257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:37.427190    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"
Oct 30 13:10:37.427257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:37.427229    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt" event={"ID":"f1a146a5-cd63-4426-a5ad-63a7018c3261","Type":"ContainerDied","Data":"09f449ab076398bd874f3da44870a7a2ef0b31410aec1eaf0fc8a73cd701d7ca"}
Oct 30 13:10:37.427508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:37.427283    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09f449ab076398bd874f3da44870a7a2ef0b31410aec1eaf0fc8a73cd701d7ca"
Oct 30 13:10:38.672413 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:10:38.672374    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:10:39.693994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.693960    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"]
Oct 30 13:10:39.694576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.694559    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f1a146a5-cd63-4426-a5ad-63a7018c3261" containerName="revoke-keys"
Oct 30 13:10:39.694576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.694576    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a146a5-cd63-4426-a5ad-63a7018c3261" containerName="revoke-keys"
Oct 30 13:10:39.694709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.694699    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a146a5-cd63-4426-a5ad-63a7018c3261" containerName="revoke-keys"
Oct 30 13:10:39.699731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.699706    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.716405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.716370    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:39.716405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.716370    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:39.717870 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.717843    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"]
Oct 30 13:10:39.739226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.736308    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:10:39.747239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.744410    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:39.804994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.804942    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.804994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.804990    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5n4s\" (UniqueName: \"kubernetes.io/projected/0fca35ed-6494-4521-b6e9-32603df6c314-kube-api-access-c5n4s\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.805267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.805174    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.906309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.906269    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.906309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.906309    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5n4s\" (UniqueName: \"kubernetes.io/projected/0fca35ed-6494-4521-b6e9-32603df6c314-kube-api-access-c5n4s\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.906591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.906372    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.907117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.907092    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.908796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.908768    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:39.932376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:39.932341    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5n4s\" (UniqueName: \"kubernetes.io/projected/0fca35ed-6494-4521-b6e9-32603df6c314-kube-api-access-c5n4s\") pod \"maas-api-revoke-sub-652e576f-7ngvm\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:40.016240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.016163    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:40.326091 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:40.326058    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fca35ed_6494_4521_b6e9_32603df6c314.slice/crio-44e0de43621a965d5439ae1822c2e6b0e6d2ed7c5d7af3c728484589a9ff2682 WatchSource:0}: Error finding container 44e0de43621a965d5439ae1822c2e6b0e6d2ed7c5d7af3c728484589a9ff2682: Status 404 returned error can't find the container with id 44e0de43621a965d5439ae1822c2e6b0e6d2ed7c5d7af3c728484589a9ff2682
Oct 30 13:10:40.329627 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.329597    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"]
Oct 30 13:10:40.464597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.464555    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm" event={"ID":"0fca35ed-6494-4521-b6e9-32603df6c314","Type":"ContainerStarted","Data":"44e0de43621a965d5439ae1822c2e6b0e6d2ed7c5d7af3c728484589a9ff2682"}
Oct 30 13:10:40.523137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.523095    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"]
Oct 30 13:10:40.528878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.528853    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.548579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.548540    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"]
Oct 30 13:10:40.616550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.616441    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.616748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.616591    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.616748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.616706    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshq5\" (UniqueName: \"kubernetes.io/projected/9e7d193e-0d98-4841-873b-afa134140a77-kube-api-access-bshq5\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.718448 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.718402    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshq5\" (UniqueName: \"kubernetes.io/projected/9e7d193e-0d98-4841-873b-afa134140a77-kube-api-access-bshq5\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.718876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.718534    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.718876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.718593    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.719236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.719182    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.721028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.721008    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.745784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.745746    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshq5\" (UniqueName: \"kubernetes.io/projected/9e7d193e-0d98-4841-873b-afa134140a77-kube-api-access-bshq5\") pod \"maas-api-revoke-sub-bab6a82f-6v79v\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:40.843470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:40.843427    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:41.160561 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:41.160520    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7d193e_0d98_4841_873b_afa134140a77.slice/crio-73d7fe25f708712eddb344c3b82ad7871a1de853cfb7f99136dcc3bef16935ff WatchSource:0}: Error finding container 73d7fe25f708712eddb344c3b82ad7871a1de853cfb7f99136dcc3bef16935ff: Status 404 returned error can't find the container with id 73d7fe25f708712eddb344c3b82ad7871a1de853cfb7f99136dcc3bef16935ff
Oct 30 13:10:41.161940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.161913    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"]
Oct 30 13:10:41.320316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.320261    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"]
Oct 30 13:10:41.325893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.325864    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.349798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.347091    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"]
Oct 30 13:10:41.448095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.447994    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.448307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.448202    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.448367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.448349    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkchn\" (UniqueName: \"kubernetes.io/projected/ccb5cb66-de36-4304-99fd-33b2e746d161-kube-api-access-nkchn\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.477009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.476968    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm" event={"ID":"0fca35ed-6494-4521-b6e9-32603df6c314","Type":"ContainerStarted","Data":"cdb404d24849353872c7f10799bbce8aa04c8ff5afec853bf31d0747b4654548"}
Oct 30 13:10:41.478548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.478517    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v" event={"ID":"9e7d193e-0d98-4841-873b-afa134140a77","Type":"ContainerStarted","Data":"73d7fe25f708712eddb344c3b82ad7871a1de853cfb7f99136dcc3bef16935ff"}
Oct 30 13:10:41.537770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.537711    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm" podStartSLOduration=2.537696199 podStartE2EDuration="2.537696199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:41.532936053 +0000 UTC m=+2306.008655662" watchObservedRunningTime="2026-10-30 13:10:41.537696199 +0000 UTC m=+2306.013415877"
Oct 30 13:10:41.549808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.549765    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.550013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.549987    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkchn\" (UniqueName: \"kubernetes.io/projected/ccb5cb66-de36-4304-99fd-33b2e746d161-kube-api-access-nkchn\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.550128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.550113    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.550524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.550505    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.552568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.552541    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.576010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.575975    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkchn\" (UniqueName: \"kubernetes.io/projected/ccb5cb66-de36-4304-99fd-33b2e746d161-kube-api-access-nkchn\") pod \"maas-api-revoke-sub-d904086b-qwnxc\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.640033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.639999    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:41.966471 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:41.966437    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb5cb66_de36_4304_99fd_33b2e746d161.slice/crio-381477b1b353927f3faea4bc7c5f6e81d7510a438ce283154ab7ccfe7ddb4bfc WatchSource:0}: Error finding container 381477b1b353927f3faea4bc7c5f6e81d7510a438ce283154ab7ccfe7ddb4bfc: Status 404 returned error can't find the container with id 381477b1b353927f3faea4bc7c5f6e81d7510a438ce283154ab7ccfe7ddb4bfc
Oct 30 13:10:41.969085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:41.969054    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"]
Oct 30 13:10:42.492185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:42.492144    4993 generic.go:356] "Generic (PLEG): container finished" podID="9e7d193e-0d98-4841-873b-afa134140a77" containerID="5a1a7400a28c7e85170903c28b7063223eef83ee741e0d9d6196ae64c7c4f36a" exitCode=0
Oct 30 13:10:42.492412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:42.492235    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v" event={"ID":"9e7d193e-0d98-4841-873b-afa134140a77","Type":"ContainerDied","Data":"5a1a7400a28c7e85170903c28b7063223eef83ee741e0d9d6196ae64c7c4f36a"}
Oct 30 13:10:42.494190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:42.494160    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc" event={"ID":"ccb5cb66-de36-4304-99fd-33b2e746d161","Type":"ContainerStarted","Data":"c87d275b517e55813b3e595456a9c00199b3522a5bdac15d7191d5acdf5fdafb"}
Oct 30 13:10:42.494365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:42.494243    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc" event={"ID":"ccb5cb66-de36-4304-99fd-33b2e746d161","Type":"ContainerStarted","Data":"381477b1b353927f3faea4bc7c5f6e81d7510a438ce283154ab7ccfe7ddb4bfc"}
Oct 30 13:10:42.496245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:42.496217    4993 generic.go:356] "Generic (PLEG): container finished" podID="0fca35ed-6494-4521-b6e9-32603df6c314" containerID="cdb404d24849353872c7f10799bbce8aa04c8ff5afec853bf31d0747b4654548" exitCode=0
Oct 30 13:10:42.496371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:42.496305    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm" event={"ID":"0fca35ed-6494-4521-b6e9-32603df6c314","Type":"ContainerDied","Data":"cdb404d24849353872c7f10799bbce8aa04c8ff5afec853bf31d0747b4654548"}
Oct 30 13:10:42.587612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:42.587503    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc" podStartSLOduration=1.5874879819999999 podStartE2EDuration="1.587487982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:42.585772837 +0000 UTC m=+2307.061492447" watchObservedRunningTime="2026-10-30 13:10:42.587487982 +0000 UTC m=+2307.063207592"
Oct 30 13:10:43.512275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:43.512235    4993 generic.go:356] "Generic (PLEG): container finished" podID="ccb5cb66-de36-4304-99fd-33b2e746d161" containerID="c87d275b517e55813b3e595456a9c00199b3522a5bdac15d7191d5acdf5fdafb" exitCode=0
Oct 30 13:10:43.512894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:43.512314    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc" event={"ID":"ccb5cb66-de36-4304-99fd-33b2e746d161","Type":"ContainerDied","Data":"c87d275b517e55813b3e595456a9c00199b3522a5bdac15d7191d5acdf5fdafb"}
Oct 30 13:10:43.925372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:43.925311    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:44.078966 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.078920    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-xks-ca\") pod \"9e7d193e-0d98-4841-873b-afa134140a77\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") "
Oct 30 13:10:44.079182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.079014    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7d193e-0d98-4841-873b-afa134140a77-kube-api-access-bshq5\" (UniqueName: \"kubernetes.io/projected/9e7d193e-0d98-4841-873b-afa134140a77-kube-api-access-bshq5\") pod \"9e7d193e-0d98-4841-873b-afa134140a77\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") "
Oct 30 13:10:44.079182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.079094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-openshift-ca\") pod \"9e7d193e-0d98-4841-873b-afa134140a77\" (UID: \"9e7d193e-0d98-4841-873b-afa134140a77\") "
Oct 30 13:10:44.184754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.184653    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-openshift-ca" pod "9e7d193e-0d98-4841-873b-afa134140a77" (UID: "9e7d193e-0d98-4841-873b-afa134140a77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:44.389231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.388409    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7d193e-0d98-4841-873b-afa134140a77-kube-api-access-bshq5" pod "9e7d193e-0d98-4841-873b-afa134140a77" (UID: "9e7d193e-0d98-4841-873b-afa134140a77"). InnerVolumeSpecName "kube-api-access-bshq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:44.442347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.411736    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-xks-ca" pod "9e7d193e-0d98-4841-873b-afa134140a77" (UID: "9e7d193e-0d98-4841-873b-afa134140a77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:44.442347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.439433    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:44.446354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.446266    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bshq5\" (UniqueName: \"kubernetes.io/projected/9e7d193e-0d98-4841-873b-afa134140a77-kube-api-access-bshq5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:44.446354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.446316    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7d193e-0d98-4841-873b-afa134140a77-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:44.767235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.766385    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:44.767235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.766937    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"
Oct 30 13:10:44.767235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.767097    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v" event={"ID":"9e7d193e-0d98-4841-873b-afa134140a77","Type":"ContainerDied","Data":"73d7fe25f708712eddb344c3b82ad7871a1de853cfb7f99136dcc3bef16935ff"}
Oct 30 13:10:44.767235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.767156    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73d7fe25f708712eddb344c3b82ad7871a1de853cfb7f99136dcc3bef16935ff"
Oct 30 13:10:44.831233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.820420    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"
Oct 30 13:10:44.831233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.820567    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm" event={"ID":"0fca35ed-6494-4521-b6e9-32603df6c314","Type":"ContainerDied","Data":"44e0de43621a965d5439ae1822c2e6b0e6d2ed7c5d7af3c728484589a9ff2682"}
Oct 30 13:10:44.831233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.820607    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44e0de43621a965d5439ae1822c2e6b0e6d2ed7c5d7af3c728484589a9ff2682"
Oct 30 13:10:44.910907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.909342    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fca35ed-6494-4521-b6e9-32603df6c314-kube-api-access-c5n4s\" (UniqueName: \"kubernetes.io/projected/0fca35ed-6494-4521-b6e9-32603df6c314-kube-api-access-c5n4s\") pod \"0fca35ed-6494-4521-b6e9-32603df6c314\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") "
Oct 30 13:10:44.910907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.909421    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-xks-ca\") pod \"0fca35ed-6494-4521-b6e9-32603df6c314\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") "
Oct 30 13:10:44.910907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.909539    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-openshift-ca\") pod \"0fca35ed-6494-4521-b6e9-32603df6c314\" (UID: \"0fca35ed-6494-4521-b6e9-32603df6c314\") "
Oct 30 13:10:44.917467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.917428    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-openshift-ca" pod "0fca35ed-6494-4521-b6e9-32603df6c314" (UID: "0fca35ed-6494-4521-b6e9-32603df6c314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:44.989232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:44.959096    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:45.026748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.026624    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:45.033230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.027742    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-xks-ca" pod "0fca35ed-6494-4521-b6e9-32603df6c314" (UID: "0fca35ed-6494-4521-b6e9-32603df6c314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:45.094045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.078160    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fca35ed-6494-4521-b6e9-32603df6c314-kube-api-access-c5n4s" pod "0fca35ed-6494-4521-b6e9-32603df6c314" (UID: "0fca35ed-6494-4521-b6e9-32603df6c314"). InnerVolumeSpecName "kube-api-access-c5n4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:45.168528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.168466    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5n4s\" (UniqueName: \"kubernetes.io/projected/0fca35ed-6494-4521-b6e9-32603df6c314-kube-api-access-c5n4s\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:45.168528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.168501    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fca35ed-6494-4521-b6e9-32603df6c314-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:45.780447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.780402    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:45.869232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.850347    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:45.869232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.863976    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc" event={"ID":"ccb5cb66-de36-4304-99fd-33b2e746d161","Type":"ContainerDied","Data":"381477b1b353927f3faea4bc7c5f6e81d7510a438ce283154ab7ccfe7ddb4bfc"}
Oct 30 13:10:45.869232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.864037    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="381477b1b353927f3faea4bc7c5f6e81d7510a438ce283154ab7ccfe7ddb4bfc"
Oct 30 13:10:45.987708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.987663    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-xks-ca\") pod \"ccb5cb66-de36-4304-99fd-33b2e746d161\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") "
Oct 30 13:10:45.988244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.988198    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb5cb66-de36-4304-99fd-33b2e746d161-kube-api-access-nkchn\" (UniqueName: \"kubernetes.io/projected/ccb5cb66-de36-4304-99fd-33b2e746d161-kube-api-access-nkchn\") pod \"ccb5cb66-de36-4304-99fd-33b2e746d161\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") "
Oct 30 13:10:45.988461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:45.988439    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-openshift-ca\") pod \"ccb5cb66-de36-4304-99fd-33b2e746d161\" (UID: \"ccb5cb66-de36-4304-99fd-33b2e746d161\") "
Oct 30 13:10:46.001499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:46.001458    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-openshift-ca" pod "ccb5cb66-de36-4304-99fd-33b2e746d161" (UID: "ccb5cb66-de36-4304-99fd-33b2e746d161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:46.101234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:46.099464    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:46.161351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:46.161292    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb5cb66-de36-4304-99fd-33b2e746d161-kube-api-access-nkchn" pod "ccb5cb66-de36-4304-99fd-33b2e746d161" (UID: "ccb5cb66-de36-4304-99fd-33b2e746d161"). InnerVolumeSpecName "kube-api-access-nkchn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:46.206282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:46.191509    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-xks-ca" pod "ccb5cb66-de36-4304-99fd-33b2e746d161" (UID: "ccb5cb66-de36-4304-99fd-33b2e746d161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:46.206282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:46.206194    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkchn\" (UniqueName: \"kubernetes.io/projected/ccb5cb66-de36-4304-99fd-33b2e746d161-kube-api-access-nkchn\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:46.206282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:46.206250    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb5cb66-de36-4304-99fd-33b2e746d161-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:46.887965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:46.887925    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"
Oct 30 13:10:51.811025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.810917    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"]
Oct 30 13:10:51.812139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812092    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9e7d193e-0d98-4841-873b-afa134140a77" containerName="revoke-keys"
Oct 30 13:10:51.812139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812139    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7d193e-0d98-4841-873b-afa134140a77" containerName="revoke-keys"
Oct 30 13:10:51.812391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812201    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ccb5cb66-de36-4304-99fd-33b2e746d161" containerName="revoke-keys"
Oct 30 13:10:51.812391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812230    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb5cb66-de36-4304-99fd-33b2e746d161" containerName="revoke-keys"
Oct 30 13:10:51.812391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812278    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0fca35ed-6494-4521-b6e9-32603df6c314" containerName="revoke-keys"
Oct 30 13:10:51.812391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812286    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fca35ed-6494-4521-b6e9-32603df6c314" containerName="revoke-keys"
Oct 30 13:10:51.812613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812514    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb5cb66-de36-4304-99fd-33b2e746d161" containerName="revoke-keys"
Oct 30 13:10:51.812613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812537    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fca35ed-6494-4521-b6e9-32603df6c314" containerName="revoke-keys"
Oct 30 13:10:51.812613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.812554    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7d193e-0d98-4841-873b-afa134140a77" containerName="revoke-keys"
Oct 30 13:10:51.817133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.817083    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:51.846547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.846505    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"]
Oct 30 13:10:51.854232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.854180    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:51.854472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.854190    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:51.854589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.854201    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:51.854589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.854317    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:10:51.894099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.894048    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54hkr\" (UniqueName: \"kubernetes.io/projected/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-kube-api-access-54hkr\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:51.894341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.894119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:51.894341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.894314    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:51.995718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.995675    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:51.995934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.995872    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54hkr\" (UniqueName: \"kubernetes.io/projected/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-kube-api-access-54hkr\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:51.995934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.995909    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:51.996684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.996653    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:51.998809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:51.998775    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:52.030697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:52.030659    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54hkr\" (UniqueName: \"kubernetes.io/projected/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-kube-api-access-54hkr\") pod \"maas-api-revoke-sub-d8880e88-hllsd\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:52.149374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:52.149276    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:52.465253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:52.465165    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"]
Oct 30 13:10:52.466136 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:52.466104    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ad446f_37bf_4a6c_b477_3f58741cb7a8.slice/crio-deaadc23db142fe42a48b14723dc58b05aaacca4e4340d8a857f8fb4eb25de67 WatchSource:0}: Error finding container deaadc23db142fe42a48b14723dc58b05aaacca4e4340d8a857f8fb4eb25de67: Status 404 returned error can't find the container with id deaadc23db142fe42a48b14723dc58b05aaacca4e4340d8a857f8fb4eb25de67
Oct 30 13:10:52.939927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:52.939888    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd" event={"ID":"a1ad446f-37bf-4a6c-b477-3f58741cb7a8","Type":"ContainerStarted","Data":"08f7a8ab4ac414e2e11d0e89ad21f39bb7b5f6608fe253a61ea08d31216386a3"}
Oct 30 13:10:52.940450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:52.939949    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd" event={"ID":"a1ad446f-37bf-4a6c-b477-3f58741cb7a8","Type":"ContainerStarted","Data":"deaadc23db142fe42a48b14723dc58b05aaacca4e4340d8a857f8fb4eb25de67"}
Oct 30 13:10:52.993791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:52.993718    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd" podStartSLOduration=1.993697565 podStartE2EDuration="1.993697565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:52.990377269 +0000 UTC m=+2317.466096880" watchObservedRunningTime="2026-10-30 13:10:52.993697565 +0000 UTC m=+2317.469417177"
Oct 30 13:10:53.960276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:53.960155    4993 generic.go:356] "Generic (PLEG): container finished" podID="a1ad446f-37bf-4a6c-b477-3f58741cb7a8" containerID="08f7a8ab4ac414e2e11d0e89ad21f39bb7b5f6608fe253a61ea08d31216386a3" exitCode=0
Oct 30 13:10:53.960764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:53.960451    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd" event={"ID":"a1ad446f-37bf-4a6c-b477-3f58741cb7a8","Type":"ContainerDied","Data":"08f7a8ab4ac414e2e11d0e89ad21f39bb7b5f6608fe253a61ea08d31216386a3"}
Oct 30 13:10:55.402172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.402135    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:55.490519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.490476    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-xks-ca\") pod \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") "
Oct 30 13:10:55.490796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.490778    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-kube-api-access-54hkr\" (UniqueName: \"kubernetes.io/projected/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-kube-api-access-54hkr\") pod \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") "
Oct 30 13:10:55.491067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.491051    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-openshift-ca\") pod \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\" (UID: \"a1ad446f-37bf-4a6c-b477-3f58741cb7a8\") "
Oct 30 13:10:55.492243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.492198    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-openshift-ca" pod "a1ad446f-37bf-4a6c-b477-3f58741cb7a8" (UID: "a1ad446f-37bf-4a6c-b477-3f58741cb7a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:55.502407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.502352    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-kube-api-access-54hkr" pod "a1ad446f-37bf-4a6c-b477-3f58741cb7a8" (UID: "a1ad446f-37bf-4a6c-b477-3f58741cb7a8"). InnerVolumeSpecName "kube-api-access-54hkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:55.505826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.505786    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-xks-ca" pod "a1ad446f-37bf-4a6c-b477-3f58741cb7a8" (UID: "a1ad446f-37bf-4a6c-b477-3f58741cb7a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:55.569938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.569826    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"]
Oct 30 13:10:55.574512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.574480    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5ba2ef-r2vgt"]
Oct 30 13:10:55.593238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.593194    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:55.593238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.593239    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54hkr\" (UniqueName: \"kubernetes.io/projected/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-kube-api-access-54hkr\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:55.593441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.593250    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad446f-37bf-4a6c-b477-3f58741cb7a8-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:10:55.696564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.696524    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a146a5-cd63-4426-a5ad-63a7018c3261" path="/var/lib/kubelet/pods/f1a146a5-cd63-4426-a5ad-63a7018c3261/volumes"
Oct 30 13:10:55.991416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.991365    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd" event={"ID":"a1ad446f-37bf-4a6c-b477-3f58741cb7a8","Type":"ContainerDied","Data":"deaadc23db142fe42a48b14723dc58b05aaacca4e4340d8a857f8fb4eb25de67"}
Oct 30 13:10:55.991416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.991395    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"
Oct 30 13:10:55.991664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:55.991474    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deaadc23db142fe42a48b14723dc58b05aaacca4e4340d8a857f8fb4eb25de67"
Oct 30 13:10:57.004523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.004474    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"]
Oct 30 13:10:57.005061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.005046    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a1ad446f-37bf-4a6c-b477-3f58741cb7a8" containerName="revoke-keys"
Oct 30 13:10:57.005120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.005062    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ad446f-37bf-4a6c-b477-3f58741cb7a8" containerName="revoke-keys"
Oct 30 13:10:57.005199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.005190    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ad446f-37bf-4a6c-b477-3f58741cb7a8" containerName="revoke-keys"
Oct 30 13:10:57.010405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.010381    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.031879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.031848    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"]
Oct 30 13:10:57.039695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.039643    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:10:57.039874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.039731    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:10:57.113808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.113776    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.114002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.113845    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.114063 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.114039    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjh98\" (UniqueName: \"kubernetes.io/projected/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-kube-api-access-qjh98\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.120070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.120046    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:10:57.120254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.120186    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:10:57.215426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.215378    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjh98\" (UniqueName: \"kubernetes.io/projected/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-kube-api-access-qjh98\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.215617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.215475    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.215617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.215557    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.216317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.216293    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.218182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.218158    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.248574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.248539    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjh98\" (UniqueName: \"kubernetes.io/projected/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-kube-api-access-qjh98\") pod \"maas-api-revoke-sub-9bd32e91-fqxvc\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.325994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.325915    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:10:57.672696 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:57.672658    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae6ca87_1e42_4c5d_b80b_8741d57892fa.slice/crio-283cd3b20347ad455295ca08f5fc08b12394fa2eb13f68bc16562a3140c92a0f WatchSource:0}: Error finding container 283cd3b20347ad455295ca08f5fc08b12394fa2eb13f68bc16562a3140c92a0f: Status 404 returned error can't find the container with id 283cd3b20347ad455295ca08f5fc08b12394fa2eb13f68bc16562a3140c92a0f
Oct 30 13:10:57.675091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:57.675066    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"]
Oct 30 13:10:58.010996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.010961    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc" event={"ID":"4ae6ca87-1e42-4c5d-b80b-8741d57892fa","Type":"ContainerStarted","Data":"283cd3b20347ad455295ca08f5fc08b12394fa2eb13f68bc16562a3140c92a0f"}
Oct 30 13:10:58.408730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.408644    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"]
Oct 30 13:10:58.413014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.412992    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.434716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.434683    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"]
Oct 30 13:10:58.533287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.533251    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.533517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.533485    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnw9l\" (UniqueName: \"kubernetes.io/projected/b45bfe02-e73a-463c-ab19-5dba0c4fde09-kube-api-access-bnw9l\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.533625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.533610    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.634863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.634819    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.635027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.634962    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnw9l\" (UniqueName: \"kubernetes.io/projected/b45bfe02-e73a-463c-ab19-5dba0c4fde09-kube-api-access-bnw9l\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.635027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.635020    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.635703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.635675    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.637558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.637529    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.663598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.663519    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnw9l\" (UniqueName: \"kubernetes.io/projected/b45bfe02-e73a-463c-ab19-5dba0c4fde09-kube-api-access-bnw9l\") pod \"maas-api-revoke-sub-bddb34a0-gjfgm\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:58.727837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:58.727796    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:10:59.020727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:59.020689    4993 generic.go:356] "Generic (PLEG): container finished" podID="4ae6ca87-1e42-4c5d-b80b-8741d57892fa" containerID="6b03e9f8d7efd9a07dfc3b409e685bc9162384314788ff236d9d02e8003dd346" exitCode=0
Oct 30 13:10:59.021153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:59.020772    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc" event={"ID":"4ae6ca87-1e42-4c5d-b80b-8741d57892fa","Type":"ContainerDied","Data":"6b03e9f8d7efd9a07dfc3b409e685bc9162384314788ff236d9d02e8003dd346"}
Oct 30 13:10:59.042544 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:10:59.042513    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45bfe02_e73a_463c_ab19_5dba0c4fde09.slice/crio-d6772135353744379458ff71f4c9bb61eb1edeacc0184798a9fbf5f8bdba0d26 WatchSource:0}: Error finding container d6772135353744379458ff71f4c9bb61eb1edeacc0184798a9fbf5f8bdba0d26: Status 404 returned error can't find the container with id d6772135353744379458ff71f4c9bb61eb1edeacc0184798a9fbf5f8bdba0d26
Oct 30 13:10:59.043817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:10:59.043792    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"]
Oct 30 13:11:00.030361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.030321    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm" event={"ID":"b45bfe02-e73a-463c-ab19-5dba0c4fde09","Type":"ContainerStarted","Data":"0819c17c7fe635e4f4138c604bc0a6bfe802429fd1bc5492acc90f1f7d72414a"}
Oct 30 13:11:00.030771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.030380    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm" event={"ID":"b45bfe02-e73a-463c-ab19-5dba0c4fde09","Type":"ContainerStarted","Data":"d6772135353744379458ff71f4c9bb61eb1edeacc0184798a9fbf5f8bdba0d26"}
Oct 30 13:11:00.079712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.079648    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm" podStartSLOduration=2.079626997 podStartE2EDuration="2.079626997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:00.074362205 +0000 UTC m=+2324.550081835" watchObservedRunningTime="2026-10-30 13:11:00.079626997 +0000 UTC m=+2324.555346608"
Oct 30 13:11:00.198876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.198845    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:11:00.258016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.257979    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-xks-ca\") pod \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") "
Oct 30 13:11:00.258316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.258283    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-kube-api-access-qjh98\" (UniqueName: \"kubernetes.io/projected/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-kube-api-access-qjh98\") pod \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") "
Oct 30 13:11:00.258466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.258392    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-openshift-ca\") pod \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\" (UID: \"4ae6ca87-1e42-4c5d-b80b-8741d57892fa\") "
Oct 30 13:11:00.259012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.258977    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-openshift-ca" pod "4ae6ca87-1e42-4c5d-b80b-8741d57892fa" (UID: "4ae6ca87-1e42-4c5d-b80b-8741d57892fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:00.268171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.268132    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-kube-api-access-qjh98" pod "4ae6ca87-1e42-4c5d-b80b-8741d57892fa" (UID: "4ae6ca87-1e42-4c5d-b80b-8741d57892fa"). InnerVolumeSpecName "kube-api-access-qjh98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:00.269722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.269686    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-xks-ca" pod "4ae6ca87-1e42-4c5d-b80b-8741d57892fa" (UID: "4ae6ca87-1e42-4c5d-b80b-8741d57892fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:00.359977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.359884    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:00.359977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.359915    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:00.359977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:00.359929    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjh98\" (UniqueName: \"kubernetes.io/projected/4ae6ca87-1e42-4c5d-b80b-8741d57892fa-kube-api-access-qjh98\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:01.040140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:01.040104    4993 generic.go:356] "Generic (PLEG): container finished" podID="b45bfe02-e73a-463c-ab19-5dba0c4fde09" containerID="0819c17c7fe635e4f4138c604bc0a6bfe802429fd1bc5492acc90f1f7d72414a" exitCode=0
Oct 30 13:11:01.040583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:01.040191    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm" event={"ID":"b45bfe02-e73a-463c-ab19-5dba0c4fde09","Type":"ContainerDied","Data":"0819c17c7fe635e4f4138c604bc0a6bfe802429fd1bc5492acc90f1f7d72414a"}
Oct 30 13:11:01.041795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:01.041775    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"
Oct 30 13:11:01.041908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:01.041771    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc" event={"ID":"4ae6ca87-1e42-4c5d-b80b-8741d57892fa","Type":"ContainerDied","Data":"283cd3b20347ad455295ca08f5fc08b12394fa2eb13f68bc16562a3140c92a0f"}
Oct 30 13:11:01.041966 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:01.041918    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="283cd3b20347ad455295ca08f5fc08b12394fa2eb13f68bc16562a3140c92a0f"
Oct 30 13:11:02.234989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.234956    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:11:02.390090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.389990    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-openshift-ca\") pod \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") "
Oct 30 13:11:02.390294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.390093    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-xks-ca\") pod \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") "
Oct 30 13:11:02.390294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.390224    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45bfe02-e73a-463c-ab19-5dba0c4fde09-kube-api-access-bnw9l\" (UniqueName: \"kubernetes.io/projected/b45bfe02-e73a-463c-ab19-5dba0c4fde09-kube-api-access-bnw9l\") pod \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\" (UID: \"b45bfe02-e73a-463c-ab19-5dba0c4fde09\") "
Oct 30 13:11:02.390500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.390472    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-openshift-ca" pod "b45bfe02-e73a-463c-ab19-5dba0c4fde09" (UID: "b45bfe02-e73a-463c-ab19-5dba0c4fde09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:02.390797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.390770    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:02.392465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.392433    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-xks-ca" pod "b45bfe02-e73a-463c-ab19-5dba0c4fde09" (UID: "b45bfe02-e73a-463c-ab19-5dba0c4fde09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:02.392562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.392486    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45bfe02-e73a-463c-ab19-5dba0c4fde09-kube-api-access-bnw9l" pod "b45bfe02-e73a-463c-ab19-5dba0c4fde09" (UID: "b45bfe02-e73a-463c-ab19-5dba0c4fde09"). InnerVolumeSpecName "kube-api-access-bnw9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:02.491659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.491616    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnw9l\" (UniqueName: \"kubernetes.io/projected/b45bfe02-e73a-463c-ab19-5dba0c4fde09-kube-api-access-bnw9l\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:02.491659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:02.491654    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45bfe02-e73a-463c-ab19-5dba0c4fde09-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:03.061963 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:03.061920    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"
Oct 30 13:11:03.062154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:03.061919    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm" event={"ID":"b45bfe02-e73a-463c-ab19-5dba0c4fde09","Type":"ContainerDied","Data":"d6772135353744379458ff71f4c9bb61eb1edeacc0184798a9fbf5f8bdba0d26"}
Oct 30 13:11:03.062154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:03.062070    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6772135353744379458ff71f4c9bb61eb1edeacc0184798a9fbf5f8bdba0d26"
Oct 30 13:11:03.625626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:03.625582    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"]
Oct 30 13:11:03.631602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:03.631560    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy" containerID="cri-o://a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691" gracePeriod=30
Oct 30 13:11:04.273913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.273867    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"]
Oct 30 13:11:04.274775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.274752    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b45bfe02-e73a-463c-ab19-5dba0c4fde09" containerName="revoke-keys"
Oct 30 13:11:04.274835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.274778    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45bfe02-e73a-463c-ab19-5dba0c4fde09" containerName="revoke-keys"
Oct 30 13:11:04.274835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.274811    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4ae6ca87-1e42-4c5d-b80b-8741d57892fa" containerName="revoke-keys"
Oct 30 13:11:04.274835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.274819    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae6ca87-1e42-4c5d-b80b-8741d57892fa" containerName="revoke-keys"
Oct 30 13:11:04.274972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.274959    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45bfe02-e73a-463c-ab19-5dba0c4fde09" containerName="revoke-keys"
Oct 30 13:11:04.275015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.274981    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae6ca87-1e42-4c5d-b80b-8741d57892fa" containerName="revoke-keys"
Oct 30 13:11:04.279386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.279362    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.301075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.301039    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"]
Oct 30 13:11:04.307493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.307438    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:11:04.307735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.307544    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:11:04.326488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.326448    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:11:04.326669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.326543    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:11:04.405704 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.405659    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"]
Oct 30 13:11:04.410151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.410125    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.429092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.429062    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"]
Oct 30 13:11:04.443486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.443448    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxzk\" (UniqueName: \"kubernetes.io/projected/0586badb-fe87-403c-badc-0ab5c95b356d-kube-api-access-mpxzk\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.443690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.443585    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.443690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.443650    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.544721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.544625    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.544887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.544812    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxzk\" (UniqueName: \"kubernetes.io/projected/0586badb-fe87-403c-badc-0ab5c95b356d-kube-api-access-mpxzk\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.544887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.544844    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.545004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.544907    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.545004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.544929    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pkd\" (UniqueName: \"kubernetes.io/projected/3511e5df-377d-4469-8c7b-50d62140cf5f-kube-api-access-r5pkd\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.545457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.545005    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.545640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.545621    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.547426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.547397    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.571799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.571759    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxzk\" (UniqueName: \"kubernetes.io/projected/0586badb-fe87-403c-badc-0ab5c95b356d-kube-api-access-mpxzk\") pod \"maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.604598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.604560    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:04.646271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.646195    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pkd\" (UniqueName: \"kubernetes.io/projected/3511e5df-377d-4469-8c7b-50d62140cf5f-kube-api-access-r5pkd\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.646742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.646379    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.646742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.646543    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.647168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.647139    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.648937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.648909    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.676399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.676319    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pkd\" (UniqueName: \"kubernetes.io/projected/3511e5df-377d-4469-8c7b-50d62140cf5f-kube-api-access-r5pkd\") pod \"maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.726479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.724462    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:04.918991 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:11:04.918950    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0586badb_fe87_403c_badc_0ab5c95b356d.slice/crio-88bc92df5c2f4bda757c7a65bb5b4749da875adf11ed3c3e1373417643c094c3 WatchSource:0}: Error finding container 88bc92df5c2f4bda757c7a65bb5b4749da875adf11ed3c3e1373417643c094c3: Status 404 returned error can't find the container with id 88bc92df5c2f4bda757c7a65bb5b4749da875adf11ed3c3e1373417643c094c3
Oct 30 13:11:04.920678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:04.920618    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"]
Oct 30 13:11:05.053006 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:11:05.052953    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3511e5df_377d_4469_8c7b_50d62140cf5f.slice/crio-92342fd0cd4fea48f796c527a9cc44143ef258945dc45d7a5c2eace45bf4c908 WatchSource:0}: Error finding container 92342fd0cd4fea48f796c527a9cc44143ef258945dc45d7a5c2eace45bf4c908: Status 404 returned error can't find the container with id 92342fd0cd4fea48f796c527a9cc44143ef258945dc45d7a5c2eace45bf4c908
Oct 30 13:11:05.055831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:05.055772    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"]
Oct 30 13:11:05.112284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:05.112244    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g" event={"ID":"0586badb-fe87-403c-badc-0ab5c95b356d","Type":"ContainerStarted","Data":"40a75284fff08459de10720c0e2cd4f02e3302d71d9febdf12b931fd2cc0401a"}
Oct 30 13:11:05.112492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:05.112308    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g" event={"ID":"0586badb-fe87-403c-badc-0ab5c95b356d","Type":"ContainerStarted","Data":"88bc92df5c2f4bda757c7a65bb5b4749da875adf11ed3c3e1373417643c094c3"}
Oct 30 13:11:05.113855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:05.113822    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz" event={"ID":"3511e5df-377d-4469-8c7b-50d62140cf5f","Type":"ContainerStarted","Data":"92342fd0cd4fea48f796c527a9cc44143ef258945dc45d7a5c2eace45bf4c908"}
Oct 30 13:11:05.161731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:05.161671    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g" podStartSLOduration=1.16165313 podStartE2EDuration="1.16165313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:05.157304149 +0000 UTC m=+2329.633023762" watchObservedRunningTime="2026-10-30 13:11:05.16165313 +0000 UTC m=+2329.637372798"
Oct 30 13:11:06.125339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:06.125201    4993 generic.go:356] "Generic (PLEG): container finished" podID="0586badb-fe87-403c-badc-0ab5c95b356d" containerID="40a75284fff08459de10720c0e2cd4f02e3302d71d9febdf12b931fd2cc0401a" exitCode=0
Oct 30 13:11:06.125339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:06.125296    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g" event={"ID":"0586badb-fe87-403c-badc-0ab5c95b356d","Type":"ContainerDied","Data":"40a75284fff08459de10720c0e2cd4f02e3302d71d9febdf12b931fd2cc0401a"}
Oct 30 13:11:06.127307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:06.127279    4993 generic.go:356] "Generic (PLEG): container finished" podID="3511e5df-377d-4469-8c7b-50d62140cf5f" containerID="91312ebbeb4ed96f9eedabb0602644cb3a38f27532dd607a8b5366a7172fee97" exitCode=0
Oct 30 13:11:06.127450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:06.127367    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz" event={"ID":"3511e5df-377d-4469-8c7b-50d62140cf5f","Type":"ContainerDied","Data":"91312ebbeb4ed96f9eedabb0602644cb3a38f27532dd607a8b5366a7172fee97"}
Oct 30 13:11:08.232285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:08.214295    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.311233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:08.282353    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.311233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:08.299953    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.526464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:08.468302    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.633233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:08.629401    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.633233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:08.629826    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.705754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:08.678460    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.888416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:08.879586    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.139580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.139452    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"]
Oct 30 13:11:09.150734 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.150708    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.163709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.163607    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:09.193660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.193621    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"]
Oct 30 13:11:09.223275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.221327    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.223275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.223140    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:11:09.266932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.266887    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.398599    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-openshift-ca\") pod \"0586badb-fe87-403c-badc-0ab5c95b356d\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") "
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.398684    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-xks-ca\") pod \"0586badb-fe87-403c-badc-0ab5c95b356d\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") "
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.398792    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0586badb-fe87-403c-badc-0ab5c95b356d-kube-api-access-mpxzk\" (UniqueName: \"kubernetes.io/projected/0586badb-fe87-403c-badc-0ab5c95b356d-kube-api-access-mpxzk\") pod \"0586badb-fe87-403c-badc-0ab5c95b356d\" (UID: \"0586badb-fe87-403c-badc-0ab5c95b356d\") "
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.398997    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.399033    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.399078    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.399104    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.399129    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aae0a50-1547-441e-9ab7-3231b86ccd73-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.399264    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8crz\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-kube-api-access-f8crz\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.399423    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.399489    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.399619    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.436822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.410719    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.495873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.476193    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 13:11:09.495873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.476559    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 13:11:09.495873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.479633    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.528001    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.528094    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.528236    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.528282    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.528333    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.528399    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.528424    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.528469    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aae0a50-1547-441e-9ab7-3231b86ccd73-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.529298    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.531077    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8crz\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-kube-api-access-f8crz\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.534661    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-openshift-ca" pod "0586badb-fe87-403c-badc-0ab5c95b356d" (UID: "0586badb-fe87-403c-badc-0ab5c95b356d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:09.566255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.563819    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aae0a50-1547-441e-9ab7-3231b86ccd73-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.567038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.566533    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.617278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.595026    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.617278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.606388    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.630613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.620052    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g" event={"ID":"0586badb-fe87-403c-badc-0ab5c95b356d","Type":"ContainerDied","Data":"88bc92df5c2f4bda757c7a65bb5b4749da875adf11ed3c3e1373417643c094c3"}
Oct 30 13:11:09.630613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.626740    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88bc92df5c2f4bda757c7a65bb5b4749da875adf11ed3c3e1373417643c094c3"
Oct 30 13:11:09.630613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.620133    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"
Oct 30 13:11:09.648266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.634667    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.648266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.635113    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 13:11:09.648266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.635716    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:11:09.648266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.635776    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 13:11:09.767303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.766405    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.767303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.767026    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 13:11:09.767303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.767576    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8crz\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-kube-api-access-f8crz\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.839778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.836954    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:09.907244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.903706    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.913241 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:11:09.909860    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:11:09.963256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.946021    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:09.963256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:09.961182    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:10.154289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.154049    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-openshift-ca\") pod \"3511e5df-377d-4469-8c7b-50d62140cf5f\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") "
Oct 30 13:11:10.154289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.154179    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3511e5df-377d-4469-8c7b-50d62140cf5f-kube-api-access-r5pkd\" (UniqueName: \"kubernetes.io/projected/3511e5df-377d-4469-8c7b-50d62140cf5f-kube-api-access-r5pkd\") pod \"3511e5df-377d-4469-8c7b-50d62140cf5f\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") "
Oct 30 13:11:10.155227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.154749    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-xks-ca\") pod \"3511e5df-377d-4469-8c7b-50d62140cf5f\" (UID: \"3511e5df-377d-4469-8c7b-50d62140cf5f\") "
Oct 30 13:11:10.156113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.155760    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-openshift-ca" pod "3511e5df-377d-4469-8c7b-50d62140cf5f" (UID: "3511e5df-377d-4469-8c7b-50d62140cf5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:10.160187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.156542    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.176080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.176008    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:10.207750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.207684    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-xks-ca" pod "3511e5df-377d-4469-8c7b-50d62140cf5f" (UID: "3511e5df-377d-4469-8c7b-50d62140cf5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:10.216514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.216449    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-xks-ca" pod "0586badb-fe87-403c-badc-0ab5c95b356d" (UID: "0586badb-fe87-403c-badc-0ab5c95b356d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:10.219835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.219393    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3511e5df-377d-4469-8c7b-50d62140cf5f-kube-api-access-r5pkd" pod "3511e5df-377d-4469-8c7b-50d62140cf5f" (UID: "3511e5df-377d-4469-8c7b-50d62140cf5f"). InnerVolumeSpecName "kube-api-access-r5pkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:10.226410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.226355    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0586badb-fe87-403c-badc-0ab5c95b356d-kube-api-access-mpxzk" pod "0586badb-fe87-403c-badc-0ab5c95b356d" (UID: "0586badb-fe87-403c-badc-0ab5c95b356d"). InnerVolumeSpecName "kube-api-access-mpxzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:10.294787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.293283    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3511e5df-377d-4469-8c7b-50d62140cf5f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.294787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.293333    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586badb-fe87-403c-badc-0ab5c95b356d-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.294787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.293350    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpxzk\" (UniqueName: \"kubernetes.io/projected/0586badb-fe87-403c-badc-0ab5c95b356d-kube-api-access-mpxzk\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.294787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.293366    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5pkd\" (UniqueName: \"kubernetes.io/projected/3511e5df-377d-4469-8c7b-50d62140cf5f-kube-api-access-r5pkd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.303723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.303693    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 13:11:10.303985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.303972    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 13:11:10.304168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.304157    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:11:10.304316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.304305    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 13:11:10.304454 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.304444    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:11:10.399446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.399398    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:10.510375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.510336    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:11:10.603309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.603269    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-credential-socket\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.603526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.603382    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istiod-ca-cert\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.603526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.603445    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-certs\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.603635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.603561    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-data\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.603635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.603592    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-kube-api-access-tgr7f\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-kube-api-access-tgr7f\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.603744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.603683    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-socket\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.604014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.603988    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-certs" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.604329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.604305    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-credential-socket" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.604527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.604500    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-data" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.604677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.604655    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istiod-ca-cert" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:10.604735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.604719    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-podinfo\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.605027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.604980    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-socket" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.605524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.605497    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-envoy\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.606190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.606159    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-token\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-token\") pod \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\" (UID: \"0cbe7dff-ab80-448b-b0d2-076e434a5cfd\") "
Oct 30 13:11:10.610180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.607409    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.610543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.610506    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.612949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.611256    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.612949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.611292    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.612949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.611306    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.612949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.612904    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-envoy" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.613643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.613612    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-kube-api-access-tgr7f" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "kube-api-access-tgr7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:10.623092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.620678    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-token" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:10.624835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.624788    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-podinfo" pod "0cbe7dff-ab80-448b-b0d2-076e434a5cfd" (UID: "0cbe7dff-ab80-448b-b0d2-076e434a5cfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:10.639633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.639602    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"
Oct 30 13:11:10.641715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.639664    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz" event={"ID":"3511e5df-377d-4469-8c7b-50d62140cf5f","Type":"ContainerDied","Data":"92342fd0cd4fea48f796c527a9cc44143ef258945dc45d7a5c2eace45bf4c908"}
Oct 30 13:11:10.641715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.639715    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92342fd0cd4fea48f796c527a9cc44143ef258945dc45d7a5c2eace45bf4c908"
Oct 30 13:11:10.643316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.643274    4993 generic.go:356] "Generic (PLEG): container finished" podID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerID="a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691" exitCode=0
Oct 30 13:11:10.644720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.644696    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"
Oct 30 13:11:10.650539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.647595    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" event={"ID":"0cbe7dff-ab80-448b-b0d2-076e434a5cfd","Type":"ContainerDied","Data":"a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691"}
Oct 30 13:11:10.650539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.647659    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl" event={"ID":"0cbe7dff-ab80-448b-b0d2-076e434a5cfd","Type":"ContainerDied","Data":"c859d57ae0f3728e7a1a38f656f5b98d4e86d7628897c19d428b472ac1ce1482"}
Oct 30 13:11:10.650539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.647687    4993 scope.go:122] "RemoveContainer" containerID="a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691"
Oct 30 13:11:10.676925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.676891    4993 scope.go:122] "RemoveContainer" containerID="3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8"
Oct 30 13:11:10.697513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.697389    4993 scope.go:122] "RemoveContainer" containerID="a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691"
Oct 30 13:11:10.697960 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:11:10.697823    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691\": container with ID starting with a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691 not found: ID does not exist" containerID="a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691"
Oct 30 13:11:10.697960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.697861    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691"} err="failed to get container status \"a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691\": rpc error: code = NotFound desc = could not find container \"a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691\": container with ID starting with a9f6e0cb54b900e3b3126f55ecf58ba0c400784f4a82e8480c710edcf79f4691 not found: ID does not exist"
Oct 30 13:11:10.697960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.697886    4993 scope.go:122] "RemoveContainer" containerID="3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8"
Oct 30 13:11:10.698406 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:11:10.698340    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8\": container with ID starting with 3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8 not found: ID does not exist" containerID="3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8"
Oct 30 13:11:10.698406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.698372    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8"} err="failed to get container status \"3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8\": rpc error: code = NotFound desc = could not find container \"3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8\": container with ID starting with 3923434a7989726847b900d37ae87d2a4a5d285d01f0995b7e78e6249595abc8 not found: ID does not exist"
Oct 30 13:11:10.714651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.714258    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgr7f\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-kube-api-access-tgr7f\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.714651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.714297    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.714651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.714315    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.714651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.714327    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbe7dff-ab80-448b-b0d2-076e434a5cfd-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.819056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.818895    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"]
Oct 30 13:11:10.824071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.822016    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-tpdvl"]
Oct 30 13:11:10.825470 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:11:10.825425    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aae0a50_1547_441e_9ab7_3231b86ccd73.slice/crio-d13c1a8aabeea7eb97461c060d4e4c35fe4dc81bcb8e36b6a1b53a732fbf3745 WatchSource:0}: Error finding container d13c1a8aabeea7eb97461c060d4e4c35fe4dc81bcb8e36b6a1b53a732fbf3745: Status 404 returned error can't find the container with id d13c1a8aabeea7eb97461c060d4e4c35fe4dc81bcb8e36b6a1b53a732fbf3745
Oct 30 13:11:10.828531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.828484    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"]
Oct 30 13:11:10.830287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.830252    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:10.830421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.830341    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:10.830421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:10.830371    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:11.684704 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:11.684663    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" event={"ID":"6aae0a50-1547-441e-9ab7-3231b86ccd73","Type":"ContainerStarted","Data":"3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566"}
Oct 30 13:11:11.685371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:11.685343    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" event={"ID":"6aae0a50-1547-441e-9ab7-3231b86ccd73","Type":"ContainerStarted","Data":"d13c1a8aabeea7eb97461c060d4e4c35fe4dc81bcb8e36b6a1b53a732fbf3745"}
Oct 30 13:11:11.702945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:11.702907    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" path="/var/lib/kubelet/pods/0cbe7dff-ab80-448b-b0d2-076e434a5cfd/volumes"
Oct 30 13:11:11.743992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:11.743939    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podStartSLOduration=3.7439223310000003 podStartE2EDuration="3.743922331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:11.742044829 +0000 UTC m=+2336.217764441" watchObservedRunningTime="2026-10-30 13:11:11.743922331 +0000 UTC m=+2336.219641919"
Oct 30 13:11:12.026166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026132    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"]
Oct 30 13:11:12.026713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026698    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy"
Oct 30 13:11:12.026774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026714    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy"
Oct 30 13:11:12.026774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026725    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy"
Oct 30 13:11:12.026774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026731    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy"
Oct 30 13:11:12.026774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026748    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3511e5df-377d-4469-8c7b-50d62140cf5f" containerName="revoke-keys"
Oct 30 13:11:12.026774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026753    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3511e5df-377d-4469-8c7b-50d62140cf5f" containerName="revoke-keys"
Oct 30 13:11:12.026774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026763    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0586badb-fe87-403c-badc-0ab5c95b356d" containerName="revoke-keys"
Oct 30 13:11:12.026774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026768    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0586badb-fe87-403c-badc-0ab5c95b356d" containerName="revoke-keys"
Oct 30 13:11:12.026989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026861    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy"
Oct 30 13:11:12.026989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026869    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe7dff-ab80-448b-b0d2-076e434a5cfd" containerName="istio-proxy"
Oct 30 13:11:12.026989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026878    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3511e5df-377d-4469-8c7b-50d62140cf5f" containerName="revoke-keys"
Oct 30 13:11:12.026989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.026883    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0586badb-fe87-403c-badc-0ab5c95b356d" containerName="revoke-keys"
Oct 30 13:11:12.030155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.030132    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.049138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.049104    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:11:12.049372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.049278    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:11:12.049599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.049577    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:11:12.054331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.054241    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"]
Oct 30 13:11:12.062257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.062173    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:11:12.135187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.135149    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv47t\" (UniqueName: \"kubernetes.io/projected/6c57e62e-8eb3-4061-b01b-4114aa6c054c-kube-api-access-xv47t\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.135447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.135413    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.135617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.135589    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.237011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.236964    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.237252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.237053    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv47t\" (UniqueName: \"kubernetes.io/projected/6c57e62e-8eb3-4061-b01b-4114aa6c054c-kube-api-access-xv47t\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.237252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.237192    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.237764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.237741    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.239749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.239722    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.288884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.288782    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv47t\" (UniqueName: \"kubernetes.io/projected/6c57e62e-8eb3-4061-b01b-4114aa6c054c-kube-api-access-xv47t\") pod \"maas-api-revoke-sub-5c981bef-nfqv7\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.345417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.345382    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:12.401175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.401141    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:12.402868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.402809    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 13:11:12.403086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.402911    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 13:11:12.701512 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:11:12.701479    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c57e62e_8eb3_4061_b01b_4114aa6c054c.slice/crio-92b85bca35f522d49e375b79ba4e774202f91c4baf24b49a1ac2eeadddab4616 WatchSource:0}: Error finding container 92b85bca35f522d49e375b79ba4e774202f91c4baf24b49a1ac2eeadddab4616: Status 404 returned error can't find the container with id 92b85bca35f522d49e375b79ba4e774202f91c4baf24b49a1ac2eeadddab4616
Oct 30 13:11:12.703291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:12.703262    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"]
Oct 30 13:11:13.400937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:13.400825    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 13:11:13.400937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:13.400888    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 13:11:13.710526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:13.710428    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7" event={"ID":"6c57e62e-8eb3-4061-b01b-4114aa6c054c","Type":"ContainerStarted","Data":"5cf05a1cad6e3a77184141089c6a3f9054030f5cdeaf4bf6261b7a72bd839541"}
Oct 30 13:11:13.710526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:13.710487    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7" event={"ID":"6c57e62e-8eb3-4061-b01b-4114aa6c054c","Type":"ContainerStarted","Data":"92b85bca35f522d49e375b79ba4e774202f91c4baf24b49a1ac2eeadddab4616"}
Oct 30 13:11:14.400616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:14.400581    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 13:11:14.400814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:14.400636    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 13:11:14.721324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:14.721194    4993 generic.go:356] "Generic (PLEG): container finished" podID="6c57e62e-8eb3-4061-b01b-4114aa6c054c" containerID="5cf05a1cad6e3a77184141089c6a3f9054030f5cdeaf4bf6261b7a72bd839541" exitCode=0
Oct 30 13:11:14.721324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:14.721281    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7" event={"ID":"6c57e62e-8eb3-4061-b01b-4114aa6c054c","Type":"ContainerDied","Data":"5cf05a1cad6e3a77184141089c6a3f9054030f5cdeaf4bf6261b7a72bd839541"}
Oct 30 13:11:15.401386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:15.401343    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 13:11:15.401585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:15.401414    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 13:11:16.215526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.214500    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:16.392775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.392726    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c57e62e-8eb3-4061-b01b-4114aa6c054c-kube-api-access-xv47t\" (UniqueName: \"kubernetes.io/projected/6c57e62e-8eb3-4061-b01b-4114aa6c054c-kube-api-access-xv47t\") pod \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") "
Oct 30 13:11:16.392956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.392805    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-openshift-ca\") pod \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") "
Oct 30 13:11:16.392956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.392912    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-xks-ca\") pod \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\" (UID: \"6c57e62e-8eb3-4061-b01b-4114aa6c054c\") "
Oct 30 13:11:16.393248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.393195    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-openshift-ca" pod "6c57e62e-8eb3-4061-b01b-4114aa6c054c" (UID: "6c57e62e-8eb3-4061-b01b-4114aa6c054c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:16.393650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.393633    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:16.395379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.395351    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c57e62e-8eb3-4061-b01b-4114aa6c054c-kube-api-access-xv47t" pod "6c57e62e-8eb3-4061-b01b-4114aa6c054c" (UID: "6c57e62e-8eb3-4061-b01b-4114aa6c054c"). InnerVolumeSpecName "kube-api-access-xv47t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:16.396281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.396259    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-xks-ca" pod "6c57e62e-8eb3-4061-b01b-4114aa6c054c" (UID: "6c57e62e-8eb3-4061-b01b-4114aa6c054c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:16.401457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.401398    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 13:11:16.401548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.401458    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 13:11:16.494570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.494528    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv47t\" (UniqueName: \"kubernetes.io/projected/6c57e62e-8eb3-4061-b01b-4114aa6c054c-kube-api-access-xv47t\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:16.494570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.494566    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c57e62e-8eb3-4061-b01b-4114aa6c054c-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:16.833567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.833523    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"
Oct 30 13:11:16.833714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.833537    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7" event={"ID":"6c57e62e-8eb3-4061-b01b-4114aa6c054c","Type":"ContainerDied","Data":"92b85bca35f522d49e375b79ba4e774202f91c4baf24b49a1ac2eeadddab4616"}
Oct 30 13:11:16.833714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:16.833668    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b85bca35f522d49e375b79ba4e774202f91c4baf24b49a1ac2eeadddab4616"
Oct 30 13:11:17.400729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:17.400689    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 13:11:17.401263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:17.400754    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 13:11:18.172311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:18.172045    4993 scope.go:122] "RemoveContainer" containerID="96f02303b28e2a212fc124bd61982671494e70224128307fc8d9a4d4d2e1bdaf"
Oct 30 13:11:18.400927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:18.400423    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 13:11:18.400927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:18.400496    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 13:11:18.855681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:18.855639    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"]
Oct 30 13:11:18.871425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:18.871369    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ffbba3-vngwz"]
Oct 30 13:11:19.502878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.502840    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"]
Oct 30 13:11:19.515846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.515803    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76c107-tjr5g"]
Oct 30 13:11:19.632358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.632328    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:19.632802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.632775    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:19.634023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.634000    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:11:19.695673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.695626    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0586badb-fe87-403c-badc-0ab5c95b356d" path="/var/lib/kubelet/pods/0586badb-fe87-403c-badc-0ab5c95b356d/volumes"
Oct 30 13:11:19.696502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.696475    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3511e5df-377d-4469-8c7b-50d62140cf5f" path="/var/lib/kubelet/pods/3511e5df-377d-4469-8c7b-50d62140cf5f/volumes"
Oct 30 13:11:19.741854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.741818    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"]
Oct 30 13:11:19.742478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.742457    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6c57e62e-8eb3-4061-b01b-4114aa6c054c" containerName="revoke-keys"
Oct 30 13:11:19.742595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.742478    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c57e62e-8eb3-4061-b01b-4114aa6c054c" containerName="revoke-keys"
Oct 30 13:11:19.742719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.742660    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c57e62e-8eb3-4061-b01b-4114aa6c054c" containerName="revoke-keys"
Oct 30 13:11:19.747757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.747733    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:19.771815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.771783    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:11:19.772138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.771910    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:11:19.773358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.773334    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"]
Oct 30 13:11:19.779345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.779295    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:11:19.779536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.779399    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:11:19.872081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.872034    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thplm\" (UniqueName: \"kubernetes.io/projected/b5d05a65-b467-4da5-853b-9097504ab51e-kube-api-access-thplm\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:19.872306 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.872187    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:19.872361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.872334    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:19.974137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.974085    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:19.974396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.974192    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:19.974396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.974372    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thplm\" (UniqueName: \"kubernetes.io/projected/b5d05a65-b467-4da5-853b-9097504ab51e-kube-api-access-thplm\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:19.974960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.974930    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:19.977634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:19.976771    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:20.000437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:20.000397    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thplm\" (UniqueName: \"kubernetes.io/projected/b5d05a65-b467-4da5-853b-9097504ab51e-kube-api-access-thplm\") pod \"maas-api-revoke-sub-3ffec38d-rtfqd\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:20.062888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:20.062795    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:20.371770 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:11:20.369891    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d05a65_b467_4da5_853b_9097504ab51e.slice/crio-534deff9f186625c654e803f042406f7a96890c2aaa56e88ef353fff2da76dcc WatchSource:0}: Error finding container 534deff9f186625c654e803f042406f7a96890c2aaa56e88ef353fff2da76dcc: Status 404 returned error can't find the container with id 534deff9f186625c654e803f042406f7a96890c2aaa56e88ef353fff2da76dcc
Oct 30 13:11:20.371770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:20.370919    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"]
Oct 30 13:11:20.872883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:20.872845    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd" event={"ID":"b5d05a65-b467-4da5-853b-9097504ab51e","Type":"ContainerStarted","Data":"cacf17609a0c8c5f45781ebfa550a0e3d4bdfa82b352550f4232b39bccd60558"}
Oct 30 13:11:20.873382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:20.872914    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd" event={"ID":"b5d05a65-b467-4da5-853b-9097504ab51e","Type":"ContainerStarted","Data":"534deff9f186625c654e803f042406f7a96890c2aaa56e88ef353fff2da76dcc"}
Oct 30 13:11:20.921536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:20.921429    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd" podStartSLOduration=1.921411684 podStartE2EDuration="1.921411684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:20.917391819 +0000 UTC m=+2345.393111441" watchObservedRunningTime="2026-10-30 13:11:20.921411684 +0000 UTC m=+2345.397131294"
Oct 30 13:11:21.884220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:21.884168    4993 generic.go:356] "Generic (PLEG): container finished" podID="b5d05a65-b467-4da5-853b-9097504ab51e" containerID="cacf17609a0c8c5f45781ebfa550a0e3d4bdfa82b352550f4232b39bccd60558" exitCode=0
Oct 30 13:11:21.884643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:21.884257    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd" event={"ID":"b5d05a65-b467-4da5-853b-9097504ab51e","Type":"ContainerDied","Data":"cacf17609a0c8c5f45781ebfa550a0e3d4bdfa82b352550f4232b39bccd60558"}
Oct 30 13:11:23.104821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.104789    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:23.220595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.220498    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-xks-ca\") pod \"b5d05a65-b467-4da5-853b-9097504ab51e\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") "
Oct 30 13:11:23.220595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.220603    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-openshift-ca\") pod \"b5d05a65-b467-4da5-853b-9097504ab51e\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") "
Oct 30 13:11:23.220849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.220671    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d05a65-b467-4da5-853b-9097504ab51e-kube-api-access-thplm\" (UniqueName: \"kubernetes.io/projected/b5d05a65-b467-4da5-853b-9097504ab51e-kube-api-access-thplm\") pod \"b5d05a65-b467-4da5-853b-9097504ab51e\" (UID: \"b5d05a65-b467-4da5-853b-9097504ab51e\") "
Oct 30 13:11:23.221043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.221010    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-openshift-ca" pod "b5d05a65-b467-4da5-853b-9097504ab51e" (UID: "b5d05a65-b467-4da5-853b-9097504ab51e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:23.221423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.221398    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:23.222800 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.222768    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-xks-ca" pod "b5d05a65-b467-4da5-853b-9097504ab51e" (UID: "b5d05a65-b467-4da5-853b-9097504ab51e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:23.222924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.222896    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d05a65-b467-4da5-853b-9097504ab51e-kube-api-access-thplm" pod "b5d05a65-b467-4da5-853b-9097504ab51e" (UID: "b5d05a65-b467-4da5-853b-9097504ab51e"). InnerVolumeSpecName "kube-api-access-thplm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:23.322994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.322949    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d05a65-b467-4da5-853b-9097504ab51e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:23.322994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.322988    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thplm\" (UniqueName: \"kubernetes.io/projected/b5d05a65-b467-4da5-853b-9097504ab51e-kube-api-access-thplm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:23.905299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.905258    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd" event={"ID":"b5d05a65-b467-4da5-853b-9097504ab51e","Type":"ContainerDied","Data":"534deff9f186625c654e803f042406f7a96890c2aaa56e88ef353fff2da76dcc"}
Oct 30 13:11:23.905480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.905322    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="534deff9f186625c654e803f042406f7a96890c2aaa56e88ef353fff2da76dcc"
Oct 30 13:11:23.905480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:23.905270    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"
Oct 30 13:11:26.672137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.672094    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"]
Oct 30 13:11:26.673061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.673032    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b5d05a65-b467-4da5-853b-9097504ab51e" containerName="revoke-keys"
Oct 30 13:11:26.673061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.673062    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d05a65-b467-4da5-853b-9097504ab51e" containerName="revoke-keys"
Oct 30 13:11:26.673270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.673252    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d05a65-b467-4da5-853b-9097504ab51e" containerName="revoke-keys"
Oct 30 13:11:26.678019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.677992    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.700537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.700490    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"]
Oct 30 13:11:26.777468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.777428    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.777669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.777574    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.777669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.777650    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.777760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.777748    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba11523-2456-4473-bb55-88f02e0bc8dc-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.777876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.777859    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.778005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.777970    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.778152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.778084    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.778152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.778130    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655vp\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-kube-api-access-655vp\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.778350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.778162    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879185    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879372    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879418    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655vp\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-kube-api-access-655vp\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879446    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879563    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879651    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879707    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879750    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba11523-2456-4473-bb55-88f02e0bc8dc-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.879905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.879859    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.880576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.880543    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.880873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.880853    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.881106 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.881087    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.882885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.882852    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.883594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.883548    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba11523-2456-4473-bb55-88f02e0bc8dc-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.884659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.884638    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.887583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.887544    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.910152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.910120    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655vp\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-kube-api-access-655vp\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.911026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.910994    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:26.998288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:26.998252    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:27.500633 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:11:27.500597    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba11523_2456_4473_bb55_88f02e0bc8dc.slice/crio-bbccf0ec23e9521015f7eb55a37eb50bcc00234fd89ac4030d89e8006b5e1707 WatchSource:0}: Error finding container bbccf0ec23e9521015f7eb55a37eb50bcc00234fd89ac4030d89e8006b5e1707: Status 404 returned error can't find the container with id bbccf0ec23e9521015f7eb55a37eb50bcc00234fd89ac4030d89e8006b5e1707
Oct 30 13:11:27.501827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:27.501795    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"]
Oct 30 13:11:27.504165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:27.504119    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:27.504301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:27.504233    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:27.504301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:27.504276    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:27.945914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:27.945811    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" event={"ID":"1ba11523-2456-4473-bb55-88f02e0bc8dc","Type":"ContainerStarted","Data":"7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd"}
Oct 30 13:11:27.945914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:27.945880    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" event={"ID":"1ba11523-2456-4473-bb55-88f02e0bc8dc","Type":"ContainerStarted","Data":"bbccf0ec23e9521015f7eb55a37eb50bcc00234fd89ac4030d89e8006b5e1707"}
Oct 30 13:11:28.007476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:28.007438    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:28.008859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:28.008831    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:28.009006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:28.008882    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:28.027250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:28.027176    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podStartSLOduration=2.027157816 podStartE2EDuration="2.027157816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:28.012854361 +0000 UTC m=+2352.488573971" watchObservedRunningTime="2026-10-30 13:11:28.027157816 +0000 UTC m=+2352.502877426"
Oct 30 13:11:29.008128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:29.008074    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:29.008705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:29.008159    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:29.513694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:29.513628    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-24.ec2.internal" podUID="fdd80fd13fc4b06ccce2af5848f9d942" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:30.007923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.007884    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:30.008116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.007950    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:30.825228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.824562    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"]
Oct 30 13:11:30.849226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.848409    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:30.873545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.870506    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:11:30.881229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.880282    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:11:30.881229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.880445    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:11:30.881229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.881073    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:11:30.918242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.893187    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"]
Oct 30 13:11:30.983572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:30.983515    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.013717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.013685    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.013947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.013935    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87skf\" (UniqueName: \"kubernetes.io/projected/3300c14c-ce34-4406-a8bc-059ec0180518-kube-api-access-87skf\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.014104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.009856    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:31.014170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.014142    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:31.115067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.114963    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.115067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.115013    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87skf\" (UniqueName: \"kubernetes.io/projected/3300c14c-ce34-4406-a8bc-059ec0180518-kube-api-access-87skf\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.115326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.115124    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.116145 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.116115    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.119625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.119593    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.144149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.144116    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87skf\" (UniqueName: \"kubernetes.io/projected/3300c14c-ce34-4406-a8bc-059ec0180518-kube-api-access-87skf\") pod \"maas-api-revoke-sub-e914f582-hmxfk\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.204485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.204433    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:31.618480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:31.614358    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"]
Oct 30 13:11:32.013352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.011740    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:32.013352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.011819    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:32.119625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.110029    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk" event={"ID":"3300c14c-ce34-4406-a8bc-059ec0180518","Type":"ContainerStarted","Data":"4825573626c5512d5242e5d05515f4da354329934dfff8d6602eb0f7d3f18096"}
Oct 30 13:11:32.719885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.719833    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"]
Oct 30 13:11:32.725437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.725401    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:32.744267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.744231    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"]
Oct 30 13:11:32.851860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.851815    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:32.852050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.851889    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:32.852050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.852028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwf9b\" (UniqueName: \"kubernetes.io/projected/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-kube-api-access-vwf9b\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:32.975285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.975164    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:32.975285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.975266    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:32.975548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.975337    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwf9b\" (UniqueName: \"kubernetes.io/projected/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-kube-api-access-vwf9b\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:32.982245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.976536    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:32.984440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:32.984407    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:33.009998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:33.005973    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwf9b\" (UniqueName: \"kubernetes.io/projected/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-kube-api-access-vwf9b\") pod \"maas-api-revoke-sub-3202a8e0-vqz9m\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:33.009998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:33.008557    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:33.009998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:33.008822    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:33.049746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:33.049710    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:33.121170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:33.121133    4993 generic.go:356] "Generic (PLEG): container finished" podID="3300c14c-ce34-4406-a8bc-059ec0180518" containerID="8d0d510d66f68838d560b1e7ca9113b6630babdbe6d5c3e8da5871ddc1e70db5" exitCode=0
Oct 30 13:11:33.121614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:33.121588    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk" event={"ID":"3300c14c-ce34-4406-a8bc-059ec0180518","Type":"ContainerDied","Data":"8d0d510d66f68838d560b1e7ca9113b6630babdbe6d5c3e8da5871ddc1e70db5"}
Oct 30 13:11:33.410062 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:11:33.409887    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3bc684_7f56_4e8d_b4d0_45ad766f8ab1.slice/crio-d3d09e84715a226e976f70ed7ee87396bab4f51100269e545670e196571b460e WatchSource:0}: Error finding container d3d09e84715a226e976f70ed7ee87396bab4f51100269e545670e196571b460e: Status 404 returned error can't find the container with id d3d09e84715a226e976f70ed7ee87396bab4f51100269e545670e196571b460e
Oct 30 13:11:33.412516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:33.412483    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"]
Oct 30 13:11:33.732240 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:11:33.722323    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3bc684_7f56_4e8d_b4d0_45ad766f8ab1.slice/crio-82591f1c61cf5b9ccced2d378ccfd260fb718b21a32a416e8911278d55775b29.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:11:33.732240 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:11:33.731283    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3bc684_7f56_4e8d_b4d0_45ad766f8ab1.slice/crio-82591f1c61cf5b9ccced2d378ccfd260fb718b21a32a416e8911278d55775b29.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:11:34.018253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:34.017725    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:34.018253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:34.017811    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:34.278234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:34.277239    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m" event={"ID":"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1","Type":"ContainerStarted","Data":"82591f1c61cf5b9ccced2d378ccfd260fb718b21a32a416e8911278d55775b29"}
Oct 30 13:11:34.278234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:34.277308    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m" event={"ID":"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1","Type":"ContainerStarted","Data":"d3d09e84715a226e976f70ed7ee87396bab4f51100269e545670e196571b460e"}
Oct 30 13:11:35.015998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:35.015876    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:35.015998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:35.015964    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:35.620149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:35.620107    4993 generic.go:356] "Generic (PLEG): container finished" podID="3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1" containerID="82591f1c61cf5b9ccced2d378ccfd260fb718b21a32a416e8911278d55775b29" exitCode=0
Oct 30 13:11:35.631490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:35.631440    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m" event={"ID":"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1","Type":"ContainerDied","Data":"82591f1c61cf5b9ccced2d378ccfd260fb718b21a32a416e8911278d55775b29"}
Oct 30 13:11:35.741519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:35.741465    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.019263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.017347    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:11:36.019263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.017422    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 13:11:36.055251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.051744    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:36.257235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.248354    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.257235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.248685    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 13:11:36.257235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.248872    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:11:36.257235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.249354    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3300c14c-ce34-4406-a8bc-059ec0180518-kube-api-access-87skf\" (UniqueName: \"kubernetes.io/projected/3300c14c-ce34-4406-a8bc-059ec0180518-kube-api-access-87skf\") pod \"3300c14c-ce34-4406-a8bc-059ec0180518\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") "
Oct 30 13:11:36.257235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.249454    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-openshift-ca\") pod \"3300c14c-ce34-4406-a8bc-059ec0180518\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") "
Oct 30 13:11:36.257235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.249554    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-xks-ca\") pod \"3300c14c-ce34-4406-a8bc-059ec0180518\" (UID: \"3300c14c-ce34-4406-a8bc-059ec0180518\") "
Oct 30 13:11:36.305730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.305687    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-openshift-ca" pod "3300c14c-ce34-4406-a8bc-059ec0180518" (UID: "3300c14c-ce34-4406-a8bc-059ec0180518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:36.308341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.308296    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.415560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.346321    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.415560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.368241    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:36.415560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.368310    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.515561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.515434    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3300c14c-ce34-4406-a8bc-059ec0180518-kube-api-access-87skf" pod "3300c14c-ce34-4406-a8bc-059ec0180518" (UID: "3300c14c-ce34-4406-a8bc-059ec0180518"). InnerVolumeSpecName "kube-api-access-87skf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:36.539451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.539385    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-xks-ca" pod "3300c14c-ce34-4406-a8bc-059ec0180518" (UID: "3300c14c-ce34-4406-a8bc-059ec0180518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:36.615246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.614367    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87skf\" (UniqueName: \"kubernetes.io/projected/3300c14c-ce34-4406-a8bc-059ec0180518-kube-api-access-87skf\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:36.615246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.614408    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3300c14c-ce34-4406-a8bc-059ec0180518-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:36.774319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.770757    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:11:36.826283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.811381    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.826283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:36.811854    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 13:11:37.150295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.142998    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"
Oct 30 13:11:37.224265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.222605    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk" event={"ID":"3300c14c-ce34-4406-a8bc-059ec0180518","Type":"ContainerDied","Data":"4825573626c5512d5242e5d05515f4da354329934dfff8d6602eb0f7d3f18096"}
Oct 30 13:11:37.224265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.222680    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4825573626c5512d5242e5d05515f4da354329934dfff8d6602eb0f7d3f18096"
Oct 30 13:11:37.224265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.223416    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.224265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.223778    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 13:11:37.256842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.256780    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 13:11:37.257647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.257603    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 30 13:11:37.259164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.257929    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 13:11:37.259430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.259414    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:11:37.277711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.277667    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.278904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.278874    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:11:37.331281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.311969    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:11:37.368806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.368407    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.369373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.369350    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 13:11:37.369595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.369580    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 13:11:37.370605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.368450    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.571254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.570130    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 13:11:37.571254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.444559    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:37.571254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.571110    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.772559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.772505    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:11:37.842316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.839799    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:11:37.938242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.777998    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 13:11:37.976186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:37.965369    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 30 13:11:38.081241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.068805    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 13:11:38.081241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.075301    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:38.098741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.092624    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 13:11:38.107670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.107632    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:38.107946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.107915    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:38.123613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.123585    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 13:11:38.132027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.131997    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 13:11:38.186186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.185964    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 13:11:38.237771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.235382    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 13:11:38.261228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.257347    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"]
Oct 30 13:11:38.323422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.323377    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3300c14c-ce34-4406-a8bc-059ec0180518" containerName="revoke-keys"
Oct 30 13:11:38.323655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.323639    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3300c14c-ce34-4406-a8bc-059ec0180518" containerName="revoke-keys"
Oct 30 13:11:38.323992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.323977    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3300c14c-ce34-4406-a8bc-059ec0180518" containerName="revoke-keys"
Oct 30 13:11:38.324829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.262081    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:38.351347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.349309    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:11:38.351347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.349358    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"]
Oct 30 13:11:38.351347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.350957    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.406991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.406956    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:11:38.421482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.421455    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:38.475780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.475737    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-openshift-ca\") pod \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") "
Oct 30 13:11:38.478560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.478526    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-xks-ca\") pod \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") "
Oct 30 13:11:38.478867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.478843    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-kube-api-access-vwf9b\" (UniqueName: \"kubernetes.io/projected/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-kube-api-access-vwf9b\") pod \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\" (UID: \"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1\") "
Oct 30 13:11:38.479385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.479358    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.479802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.479779    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5ws\" (UniqueName: \"kubernetes.io/projected/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-kube-api-access-nm5ws\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.479961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.479946    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.481150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.478339    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-openshift-ca" pod "3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1" (UID: "3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:38.609326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.605775    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5ws\" (UniqueName: \"kubernetes.io/projected/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-kube-api-access-nm5ws\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.609326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.605849    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.609326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.606035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.609326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.606131    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:38.609326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.606998    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.620087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.618449    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-xks-ca" pod "3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1" (UID: "3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:38.671249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.665565    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-kube-api-access-vwf9b" pod "3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1" (UID: "3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1"). InnerVolumeSpecName "kube-api-access-vwf9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:38.697258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.695383    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:38.709792 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:11:38.706998    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:11:38.717934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.717461    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:38.717934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.717500    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwf9b\" (UniqueName: \"kubernetes.io/projected/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1-kube-api-access-vwf9b\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:38.747195 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:38.747156    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5ws\" (UniqueName: \"kubernetes.io/projected/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-kube-api-access-nm5ws\") pod \"maas-api-revoke-sub-eeaf4097-m26wr\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:39.034553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.034510    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:39.133449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.131090    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:39.133449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.131150    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:39.268643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.268612    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"
Oct 30 13:11:39.268908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.268635    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m" event={"ID":"3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1","Type":"ContainerDied","Data":"d3d09e84715a226e976f70ed7ee87396bab4f51100269e545670e196571b460e"}
Oct 30 13:11:39.269060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.268941    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3d09e84715a226e976f70ed7ee87396bab4f51100269e545670e196571b460e"
Oct 30 13:11:39.383194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.383083    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"]
Oct 30 13:11:39.384012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.383983    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1" containerName="revoke-keys"
Oct 30 13:11:39.384012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.384009    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1" containerName="revoke-keys"
Oct 30 13:11:39.384261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.384233    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1" containerName="revoke-keys"
Oct 30 13:11:39.389840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.389803    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.420121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.420016    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"]
Oct 30 13:11:39.428412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.426278    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"]
Oct 30 13:11:39.434509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.434473    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.434689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.434665    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.434794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.434774    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjk8\" (UniqueName: \"kubernetes.io/projected/311b122f-7dbb-4986-8972-bb799bc94830-kube-api-access-5zjk8\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.536114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.536061    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.536349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.536243    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.536412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.536356    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjk8\" (UniqueName: \"kubernetes.io/projected/311b122f-7dbb-4986-8972-bb799bc94830-kube-api-access-5zjk8\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.537480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.537447    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.538888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.538861    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.564199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.564159    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjk8\" (UniqueName: \"kubernetes.io/projected/311b122f-7dbb-4986-8972-bb799bc94830-kube-api-access-5zjk8\") pod \"maas-api-revoke-sub-5722bbb7-6xxcd\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:39.709304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:39.709256    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:40.048596 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:11:40.048560    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311b122f_7dbb_4986_8972_bb799bc94830.slice/crio-893d7df9d4ce4f0652019624d133108cc60376928bfb967b30af5ec48ea7b948 WatchSource:0}: Error finding container 893d7df9d4ce4f0652019624d133108cc60376928bfb967b30af5ec48ea7b948: Status 404 returned error can't find the container with id 893d7df9d4ce4f0652019624d133108cc60376928bfb967b30af5ec48ea7b948
Oct 30 13:11:40.050264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.050236    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"]
Oct 30 13:11:40.179509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.179390    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:40.179714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.179522    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:40.254607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.254568    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:40.255719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.255435    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:40.256423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.256395    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:11:40.284433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.284388    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd" event={"ID":"311b122f-7dbb-4986-8972-bb799bc94830","Type":"ContainerStarted","Data":"893d7df9d4ce4f0652019624d133108cc60376928bfb967b30af5ec48ea7b948"}
Oct 30 13:11:40.286498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.286458    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr" event={"ID":"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e","Type":"ContainerStarted","Data":"3a6e6850f70226d50bb64bfd5418a988bce8626be7c6c1639db65349e35bbf8c"}
Oct 30 13:11:40.286657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.286526    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr" event={"ID":"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e","Type":"ContainerStarted","Data":"e44751cca99f5701e15f2ce600d62c816b6bb278552046e7f3364c5a0dd65a8f"}
Oct 30 13:11:40.414407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:40.414337    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr" podStartSLOduration=3.414316678 podStartE2EDuration="3.414316678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:40.408195796 +0000 UTC m=+2364.883915409" watchObservedRunningTime="2026-10-30 13:11:40.414316678 +0000 UTC m=+2364.890036289"
Oct 30 13:11:40.677826 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:11:40.677777    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c75d8e_c134_48a8_9a4c_9e7a47a2bc1e.slice/crio-conmon-3a6e6850f70226d50bb64bfd5418a988bce8626be7c6c1639db65349e35bbf8c.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:11:41.300045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:41.300007    4993 generic.go:356] "Generic (PLEG): container finished" podID="43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e" containerID="3a6e6850f70226d50bb64bfd5418a988bce8626be7c6c1639db65349e35bbf8c" exitCode=0
Oct 30 13:11:41.300507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:41.300091    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr" event={"ID":"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e","Type":"ContainerDied","Data":"3a6e6850f70226d50bb64bfd5418a988bce8626be7c6c1639db65349e35bbf8c"}
Oct 30 13:11:41.302417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:41.302387    4993 generic.go:356] "Generic (PLEG): container finished" podID="311b122f-7dbb-4986-8972-bb799bc94830" containerID="a3c0c1c62f87426677837e3508d0c4df70f007c5729c787b2545af5fefdb370a" exitCode=0
Oct 30 13:11:41.302584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:41.302461    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd" event={"ID":"311b122f-7dbb-4986-8972-bb799bc94830","Type":"ContainerDied","Data":"a3c0c1c62f87426677837e3508d0c4df70f007c5729c787b2545af5fefdb370a"}
Oct 30 13:11:42.517459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.517427    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:42.553340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.553309    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:42.595334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.593150    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-openshift-ca\") pod \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") "
Oct 30 13:11:42.595334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.593263    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-kube-api-access-nm5ws\" (UniqueName: \"kubernetes.io/projected/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-kube-api-access-nm5ws\") pod \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") "
Oct 30 13:11:42.595334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.593326    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311b122f-7dbb-4986-8972-bb799bc94830-kube-api-access-5zjk8\" (UniqueName: \"kubernetes.io/projected/311b122f-7dbb-4986-8972-bb799bc94830-kube-api-access-5zjk8\") pod \"311b122f-7dbb-4986-8972-bb799bc94830\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") "
Oct 30 13:11:42.595334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.593451    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-xks-ca\") pod \"311b122f-7dbb-4986-8972-bb799bc94830\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") "
Oct 30 13:11:42.595334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.593491    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-xks-ca\") pod \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\" (UID: \"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e\") "
Oct 30 13:11:42.595334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.593522    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-openshift-ca\") pod \"311b122f-7dbb-4986-8972-bb799bc94830\" (UID: \"311b122f-7dbb-4986-8972-bb799bc94830\") "
Oct 30 13:11:42.595334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.594554    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-openshift-ca" pod "311b122f-7dbb-4986-8972-bb799bc94830" (UID: "311b122f-7dbb-4986-8972-bb799bc94830"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:42.624696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.624652    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-openshift-ca" pod "43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e" (UID: "43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:42.630541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.630499    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-xks-ca" pod "311b122f-7dbb-4986-8972-bb799bc94830" (UID: "311b122f-7dbb-4986-8972-bb799bc94830"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:42.642332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.642281    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311b122f-7dbb-4986-8972-bb799bc94830-kube-api-access-5zjk8" pod "311b122f-7dbb-4986-8972-bb799bc94830" (UID: "311b122f-7dbb-4986-8972-bb799bc94830"). InnerVolumeSpecName "kube-api-access-5zjk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:42.642748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.642723    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-kube-api-access-nm5ws" pod "43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e" (UID: "43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e"). InnerVolumeSpecName "kube-api-access-nm5ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:42.646675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.646639    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-xks-ca" pod "43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e" (UID: "43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:42.695861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.695818    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjk8\" (UniqueName: \"kubernetes.io/projected/311b122f-7dbb-4986-8972-bb799bc94830-kube-api-access-5zjk8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:42.695861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.695857    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:42.695861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.695869    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:42.696119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.695877    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b122f-7dbb-4986-8972-bb799bc94830-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:42.696119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.695886    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:42.696119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:42.695895    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm5ws\" (UniqueName: \"kubernetes.io/projected/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e-kube-api-access-nm5ws\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:11:43.321033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:43.321005    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"
Oct 30 13:11:43.321276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:43.321025    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd" event={"ID":"311b122f-7dbb-4986-8972-bb799bc94830","Type":"ContainerDied","Data":"893d7df9d4ce4f0652019624d133108cc60376928bfb967b30af5ec48ea7b948"}
Oct 30 13:11:43.321276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:43.321078    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="893d7df9d4ce4f0652019624d133108cc60376928bfb967b30af5ec48ea7b948"
Oct 30 13:11:43.322563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:43.322539    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"
Oct 30 13:11:43.322673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:43.322564    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr" event={"ID":"43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e","Type":"ContainerDied","Data":"e44751cca99f5701e15f2ce600d62c816b6bb278552046e7f3364c5a0dd65a8f"}
Oct 30 13:11:43.322673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:43.322606    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44751cca99f5701e15f2ce600d62c816b6bb278552046e7f3364c5a0dd65a8f"
Oct 30 13:11:54.189191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:54.189152    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"]
Oct 30 13:11:54.190061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:54.190024    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy" containerID="cri-o://eb366daadaec2e3c89fcc0454265375a9d2df7b5065391a206e7b2aa081840d8" gracePeriod=30
Oct 30 13:11:57.852250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:57.848348    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:58.001233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:58.000514    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:58.131293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:58.130334    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:58.197233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:58.190308    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:58.383291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:58.367346    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:58.414234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:58.413546    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:58.870234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:58.851305    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:11:58.870234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:58.852255    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 13:11:59.116074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.114858    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.162785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.162687    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.165253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.163380    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 13:11:59.180890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.162592    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 13:11:59.180890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.172753    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 13:11:59.214155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.214110    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.214812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.214784    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 13:11:59.388811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.386352    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.388811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.387186    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:11:59.434539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.434408    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.853093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.853025    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.958280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:11:59.957960    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 13:12:00.176103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:00.175998    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:12:00.190423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:00.190383    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 13:12:00.200964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:00.200911    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 13:12:00.211346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:00.206712    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:00.477313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:00.476378    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:12:00.562230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:00.545589    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 13:12:01.169407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.111953    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 13:12:01.262353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.150892    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:01.262353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.202572    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:01.328186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.328154    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:12:01.450929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.450827    4993 generic.go:356] "Generic (PLEG): container finished" podID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerID="eb366daadaec2e3c89fcc0454265375a9d2df7b5065391a206e7b2aa081840d8" exitCode=0
Oct 30 13:12:01.451767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.451740    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" event={"ID":"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5","Type":"ContainerDied","Data":"eb366daadaec2e3c89fcc0454265375a9d2df7b5065391a206e7b2aa081840d8"}
Oct 30 13:12:01.470227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.464676    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 13:12:01.539272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.539239    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 13:12:01.717577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717467    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-kube-api-access-t899q\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-kube-api-access-t899q\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.717577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717548    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-socket\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.717908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717602    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-certs\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.717908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717635    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-data\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.717908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717667    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-envoy\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.717908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717708    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-podinfo\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.717908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717773    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-token\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-token\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.717908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717882    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-credential-socket\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.717908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.717883    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-socket" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.718339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.718145    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-certs" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.718339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.718266    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istiod-ca-cert\") pod \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\" (UID: \"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5\") "
Oct 30 13:12:01.718801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.718599    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-data" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.718923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.718869    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.718923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.718891    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.718923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.718906    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.719179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.719148    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-credential-socket" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.719353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.719193    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istiod-ca-cert" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:01.726343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.726302    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-podinfo" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:01.726612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.726419    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-token" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:01.726841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.726781    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-envoy" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.727611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.727583    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-kube-api-access-t899q" pod "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" (UID: "6b6e4240-e2f8-4632-89b3-bfb1c1610ec5"). InnerVolumeSpecName "kube-api-access-t899q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:01.820674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.820633    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.820674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.820672    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.820912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.820689    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.820912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.820700    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t899q\" (UniqueName: \"kubernetes.io/projected/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-kube-api-access-t899q\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.820912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.820709    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.820912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:01.820719    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.464891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:02.464863    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"
Oct 30 13:12:02.465382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:02.464855    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb" event={"ID":"6b6e4240-e2f8-4632-89b3-bfb1c1610ec5","Type":"ContainerDied","Data":"0988efe6827bfd8f4d957d228a6f776735373b74a15c359bbd145faf77123f5f"}
Oct 30 13:12:02.465382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:02.465015    4993 scope.go:122] "RemoveContainer" containerID="eb366daadaec2e3c89fcc0454265375a9d2df7b5065391a206e7b2aa081840d8"
Oct 30 13:12:02.632004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:02.631691    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"]
Oct 30 13:12:02.635893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:02.635851    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-nc8rb"]
Oct 30 13:12:03.693582 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:03.693548    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" path="/var/lib/kubelet/pods/6b6e4240-e2f8-4632-89b3-bfb1c1610ec5/volumes"
Oct 30 13:12:07.224873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.224832    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"]
Oct 30 13:12:07.225629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225608    4993 cpu_manager.go:414] "containerMap: removing container" podUID="311b122f-7dbb-4986-8972-bb799bc94830" containerName="revoke-keys"
Oct 30 13:12:07.225728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225630    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311b122f-7dbb-4986-8972-bb799bc94830" containerName="revoke-keys"
Oct 30 13:12:07.225728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225665    4993 cpu_manager.go:414] "containerMap: removing container" podUID="43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e" containerName="revoke-keys"
Oct 30 13:12:07.225728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225672    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e" containerName="revoke-keys"
Oct 30 13:12:07.225728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225686    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy"
Oct 30 13:12:07.225728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225693    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy"
Oct 30 13:12:07.225977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225865    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="311b122f-7dbb-4986-8972-bb799bc94830" containerName="revoke-keys"
Oct 30 13:12:07.225977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225884    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e" containerName="revoke-keys"
Oct 30 13:12:07.225977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.225896    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6e4240-e2f8-4632-89b3-bfb1c1610ec5" containerName="istio-proxy"
Oct 30 13:12:07.230792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.230768    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.251174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.251142    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"]
Oct 30 13:12:07.256643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.256441    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt27x\" (UniqueName: \"kubernetes.io/projected/0b81dd24-27d8-46fc-90ad-f68d33d9e810-kube-api-access-tt27x\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.256643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.256537    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.256862 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.256680    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.257917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.257887    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:12:07.258047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.257951    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:12:07.258047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.258009    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:12:07.261397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.261376    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:12:07.357866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.357831    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt27x\" (UniqueName: \"kubernetes.io/projected/0b81dd24-27d8-46fc-90ad-f68d33d9e810-kube-api-access-tt27x\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.358085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.357896    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.358085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.358049    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.358590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.358565    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.360468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.360444    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.382566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.382540    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt27x\" (UniqueName: \"kubernetes.io/projected/0b81dd24-27d8-46fc-90ad-f68d33d9e810-kube-api-access-tt27x\") pod \"maas-api-revoke-sub-e19a9414-wgd2p\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.549379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.548028    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:07.929576 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:12:07.929544    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b81dd24_27d8_46fc_90ad_f68d33d9e810.slice/crio-33a1fdb961b7069369e00d5c7b1fb3e11c5947d0cbb97fb5dd285acd368ca31e WatchSource:0}: Error finding container 33a1fdb961b7069369e00d5c7b1fb3e11c5947d0cbb97fb5dd285acd368ca31e: Status 404 returned error can't find the container with id 33a1fdb961b7069369e00d5c7b1fb3e11c5947d0cbb97fb5dd285acd368ca31e
Oct 30 13:12:07.931874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:07.931849    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"]
Oct 30 13:12:08.570168 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:12:08.570123    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:12:08.572700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:08.572665    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p" event={"ID":"0b81dd24-27d8-46fc-90ad-f68d33d9e810","Type":"ContainerStarted","Data":"5c0f99651478b100c1d35ed5fea7d74a158de29b7d4f331db5dc028bd7107245"}
Oct 30 13:12:08.572825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:08.572733    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p" event={"ID":"0b81dd24-27d8-46fc-90ad-f68d33d9e810","Type":"ContainerStarted","Data":"33a1fdb961b7069369e00d5c7b1fb3e11c5947d0cbb97fb5dd285acd368ca31e"}
Oct 30 13:12:08.620501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:08.620386    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p" podStartSLOduration=1.620367897 podStartE2EDuration="1.620367897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:08.617136293 +0000 UTC m=+2393.092855906" watchObservedRunningTime="2026-10-30 13:12:08.620367897 +0000 UTC m=+2393.096087508"
Oct 30 13:12:09.583794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:09.583758    4993 generic.go:356] "Generic (PLEG): container finished" podID="0b81dd24-27d8-46fc-90ad-f68d33d9e810" containerID="5c0f99651478b100c1d35ed5fea7d74a158de29b7d4f331db5dc028bd7107245" exitCode=0
Oct 30 13:12:09.584271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:09.583879    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p" event={"ID":"0b81dd24-27d8-46fc-90ad-f68d33d9e810","Type":"ContainerDied","Data":"5c0f99651478b100c1d35ed5fea7d74a158de29b7d4f331db5dc028bd7107245"}
Oct 30 13:12:10.775814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.775785    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:10.859012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.858907    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b81dd24-27d8-46fc-90ad-f68d33d9e810-kube-api-access-tt27x\" (UniqueName: \"kubernetes.io/projected/0b81dd24-27d8-46fc-90ad-f68d33d9e810-kube-api-access-tt27x\") pod \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") "
Oct 30 13:12:10.859012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.858953    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-xks-ca\") pod \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") "
Oct 30 13:12:10.859012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.858976    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-openshift-ca\") pod \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\" (UID: \"0b81dd24-27d8-46fc-90ad-f68d33d9e810\") "
Oct 30 13:12:10.859465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.859433    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-openshift-ca" pod "0b81dd24-27d8-46fc-90ad-f68d33d9e810" (UID: "0b81dd24-27d8-46fc-90ad-f68d33d9e810"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:10.859649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.859534    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:10.861357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.861324    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b81dd24-27d8-46fc-90ad-f68d33d9e810-kube-api-access-tt27x" pod "0b81dd24-27d8-46fc-90ad-f68d33d9e810" (UID: "0b81dd24-27d8-46fc-90ad-f68d33d9e810"). InnerVolumeSpecName "kube-api-access-tt27x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:10.861641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.861620    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-xks-ca" pod "0b81dd24-27d8-46fc-90ad-f68d33d9e810" (UID: "0b81dd24-27d8-46fc-90ad-f68d33d9e810"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:10.960624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.960586    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt27x\" (UniqueName: \"kubernetes.io/projected/0b81dd24-27d8-46fc-90ad-f68d33d9e810-kube-api-access-tt27x\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:10.960624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:10.960618    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b81dd24-27d8-46fc-90ad-f68d33d9e810-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:11.605115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:11.605084    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"
Oct 30 13:12:11.605115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:11.605108    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p" event={"ID":"0b81dd24-27d8-46fc-90ad-f68d33d9e810","Type":"ContainerDied","Data":"33a1fdb961b7069369e00d5c7b1fb3e11c5947d0cbb97fb5dd285acd368ca31e"}
Oct 30 13:12:11.605394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:11.605156    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a1fdb961b7069369e00d5c7b1fb3e11c5947d0cbb97fb5dd285acd368ca31e"
Oct 30 13:12:16.101337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:16.101301    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:12:16.115516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:16.115486    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:12:18.282563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:18.282526    4993 scope.go:122] "RemoveContainer" containerID="03e8ff84aaf2a8ecdf5ece47b0fc0f345f37c9578150ed30a4044bd609185e04"
Oct 30 13:12:18.310081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:18.310055    4993 scope.go:122] "RemoveContainer" containerID="2530da77b85879bbbab288013e5d03291a6f9a7f0f3f62d1db4131a52f04c2bb"
Oct 30 13:12:18.351454 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:18.351424    4993 scope.go:122] "RemoveContainer" containerID="c54d6e69d42d9d3bcd7fea62e634206cf600166ea364440799fc6df8ab702a1c"
Oct 30 13:12:18.366221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:18.366186    4993 scope.go:122] "RemoveContainer" containerID="6bfc4d6118ce55fd5bb08fab87c577a63b8bbbb7c62a0e363e4d784872e0baab"
Oct 30 13:12:18.384340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:18.384308    4993 scope.go:122] "RemoveContainer" containerID="817bd2206440cb0816573667bb6111eb173436dc90630217daade169002cac80"
Oct 30 13:12:38.609298 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:12:38.609199    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:12:46.706907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.706862    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"]
Oct 30 13:12:46.707973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.707946    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0b81dd24-27d8-46fc-90ad-f68d33d9e810" containerName="revoke-keys"
Oct 30 13:12:46.708094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.707973    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b81dd24-27d8-46fc-90ad-f68d33d9e810" containerName="revoke-keys"
Oct 30 13:12:46.708266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.708244    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b81dd24-27d8-46fc-90ad-f68d33d9e810" containerName="revoke-keys"
Oct 30 13:12:46.712901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.712879    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.736413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.736377    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"]
Oct 30 13:12:46.748476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.748442    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-241ffa-openshift-default-dockercfg-nw684\""
Oct 30 13:12:46.784670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.784627    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-certs\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.784871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.784709    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12584623-fe58-48cb-b84c-2dc5012a7a7d-istiod-ca-cert\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.784871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.784753    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-podinfo\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.784871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.784830    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmv9\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-kube-api-access-4dmv9\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.785061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.784880    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-data\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.785061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.784955    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-credential-socket\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.785061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.784984    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-socket\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.785061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.785050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-envoy\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.785347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.785180    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-token\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886250    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-envoy\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886351    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-token\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886394    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-certs\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886452    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12584623-fe58-48cb-b84c-2dc5012a7a7d-istiod-ca-cert\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886492    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-podinfo\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886554    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmv9\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-kube-api-access-4dmv9\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886601    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-data\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886671    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-credential-socket\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.886749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.886701    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-socket\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.887344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.887312    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12584623-fe58-48cb-b84c-2dc5012a7a7d-istiod-ca-cert\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.887806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.887728    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-certs\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.888266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.888188    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-credential-socket\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.888370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.888289    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-data\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.888651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.888623    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-socket\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.889558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.889533    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-envoy\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.889666 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.889580    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-podinfo\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.930786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.930715    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-token\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:46.931918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:46.931885    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmv9\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-kube-api-access-4dmv9\") pod \"e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") " pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:47.044232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:47.043470    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:47.540695 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:12:47.540657    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12584623_fe58_48cb_b84c_2dc5012a7a7d.slice/crio-8bcfbf1f598a68b90eab2aa83459e291f2c4b471d69ba0b083d64aa93c088f24 WatchSource:0}: Error finding container 8bcfbf1f598a68b90eab2aa83459e291f2c4b471d69ba0b083d64aa93c088f24: Status 404 returned error can't find the container with id 8bcfbf1f598a68b90eab2aa83459e291f2c4b471d69ba0b083d64aa93c088f24
Oct 30 13:12:47.543346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:47.543312    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"]
Oct 30 13:12:47.543703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:47.543664    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:12:47.543770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:47.543753    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:12:47.543815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:47.543800    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:12:48.225229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:48.225060    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" event={"ID":"12584623-fe58-48cb-b84c-2dc5012a7a7d","Type":"ContainerStarted","Data":"bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72"}
Oct 30 13:12:48.225229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:48.225167    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" event={"ID":"12584623-fe58-48cb-b84c-2dc5012a7a7d","Type":"ContainerStarted","Data":"8bcfbf1f598a68b90eab2aa83459e291f2c4b471d69ba0b083d64aa93c088f24"}
Oct 30 13:12:48.333393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:48.333327    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podStartSLOduration=2.333308707 podStartE2EDuration="2.333308707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:48.329600026 +0000 UTC m=+2432.805319662" watchObservedRunningTime="2026-10-30 13:12:48.333308707 +0000 UTC m=+2432.809028354"
Oct 30 13:12:49.047624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:49.047590    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:49.059283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:49.059238    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:49.233867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:49.233833    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:49.234765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:49.234746    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:12:54.365835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:54.365779    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"]
Oct 30 13:12:54.366400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:54.366077    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy" containerID="cri-o://f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f" gracePeriod=30
Oct 30 13:12:59.655259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.655223    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 13:12:59.711413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711367    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-token\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-token\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.711620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711447    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-data\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.711620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711553    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8723b3c3-b64e-483f-9efb-c092aa90aa34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8723b3c3-b64e-483f-9efb-c092aa90aa34-istiod-ca-cert\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.711742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711667    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-kube-api-access-rxhr4\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-kube-api-access-rxhr4\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.711742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711709    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-envoy\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.711860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711749    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-credential-socket\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.711860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711790    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-podinfo\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.711860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711795    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-data" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:59.711860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711833    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-socket\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.712108 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.711915    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-certs\") pod \"8723b3c3-b64e-483f-9efb-c092aa90aa34\" (UID: \"8723b3c3-b64e-483f-9efb-c092aa90aa34\") "
Oct 30 13:12:59.712588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.712562    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:59.712760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.712653    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8723b3c3-b64e-483f-9efb-c092aa90aa34-istiod-ca-cert" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:59.712923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.712861    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-socket" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:59.713091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.713064    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-certs" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:59.713159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.713097    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-credential-socket" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:59.714101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.714068    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-token" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:59.714240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.714195    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-kube-api-access-rxhr4" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "kube-api-access-rxhr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:59.714298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.714253    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-podinfo" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:59.714298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.714272    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-envoy" pod "8723b3c3-b64e-483f-9efb-c092aa90aa34" (UID: "8723b3c3-b64e-483f-9efb-c092aa90aa34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:59.814507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.814464    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:59.814507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.814498    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:59.814507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.814508    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8723b3c3-b64e-483f-9efb-c092aa90aa34-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:59.814789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.814518    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxhr4\" (UniqueName: \"kubernetes.io/projected/8723b3c3-b64e-483f-9efb-c092aa90aa34-kube-api-access-rxhr4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:59.814789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.814529    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:59.814789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.814538    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:59.814789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.814548    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8723b3c3-b64e-483f-9efb-c092aa90aa34-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:12:59.814789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:12:59.814558    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8723b3c3-b64e-483f-9efb-c092aa90aa34-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:13:00.337860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.337824    4993 generic.go:356] "Generic (PLEG): container finished" podID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerID="f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f" exitCode=0
Oct 30 13:13:00.338071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.337881    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"
Oct 30 13:13:00.338071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.337907    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" event={"ID":"8723b3c3-b64e-483f-9efb-c092aa90aa34","Type":"ContainerDied","Data":"f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f"}
Oct 30 13:13:00.338071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.337975    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs" event={"ID":"8723b3c3-b64e-483f-9efb-c092aa90aa34","Type":"ContainerDied","Data":"6d37e980bc72e0bf28a1ecdcb13a6792f87e3a1bec0976442943df197790bc2f"}
Oct 30 13:13:00.338071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.338006    4993 scope.go:122] "RemoveContainer" containerID="f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f"
Oct 30 13:13:00.370634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.370605    4993 scope.go:122] "RemoveContainer" containerID="f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f"
Oct 30 13:13:00.370983 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:13:00.370961    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f\": container with ID starting with f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f not found: ID does not exist" containerID="f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f"
Oct 30 13:13:00.371067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.370996    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f"} err="failed to get container status \"f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f\": rpc error: code = NotFound desc = could not find container \"f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f\": container with ID starting with f1a314285e3facbae6e1eb98e752142962a4e8bcbfb0f4d38228a7c56df8779f not found: ID does not exist"
Oct 30 13:13:00.447391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.447344    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"]
Oct 30 13:13:00.463174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:00.463133    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-zzxjs"]
Oct 30 13:13:01.692710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:01.692675    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" path="/var/lib/kubelet/pods/8723b3c3-b64e-483f-9efb-c092aa90aa34/volumes"
Oct 30 13:13:08.574809 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:13:08.574765    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:13:15.007101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.007059    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"]
Oct 30 13:13:15.007941 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.007917    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy"
Oct 30 13:13:15.008067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.007942    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy"
Oct 30 13:13:15.008125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.008108    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8723b3c3-b64e-483f-9efb-c092aa90aa34" containerName="istio-proxy"
Oct 30 13:13:15.025830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.025800    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.044835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.044805    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"]
Oct 30 13:13:15.047723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.047696    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-71fbeb-openshift-default-dockercfg-wkfpw\""
Oct 30 13:13:15.078177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.078131    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79100f5f-7848-488a-819a-eb9227e0e055-istiod-ca-cert\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.078549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.078517    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-istio-token\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.078728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.078707    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-certs\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.078877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.078860    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-socket\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.079019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.079002    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-credential-socket\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.079364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.079339    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-data\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.079593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.079572    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-envoy\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.079843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.079822    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f64qq\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-kube-api-access-f64qq\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.079994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.079975    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79100f5f-7848-488a-819a-eb9227e0e055-istio-podinfo\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183225    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-envoy\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183510    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64qq\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-kube-api-access-f64qq\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183577    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79100f5f-7848-488a-819a-eb9227e0e055-istio-podinfo\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183658    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79100f5f-7848-488a-819a-eb9227e0e055-istiod-ca-cert\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183740    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-istio-token\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183786    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-certs\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183819    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-socket\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183853    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-credential-socket\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.183895    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-data\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.184892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.184431    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-data\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.233583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.209617    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-certs\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.233583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.209888    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-socket\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.233583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.210090    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-credential-socket\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.233583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.210310    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-envoy\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.233583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.211306    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79100f5f-7848-488a-819a-eb9227e0e055-istiod-ca-cert\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.261133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.261048    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79100f5f-7848-488a-819a-eb9227e0e055-istio-podinfo\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.274056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.274019    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f64qq\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-kube-api-access-f64qq\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.274895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.274870    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-istio-token\") pod \"e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") " pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.353233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.352879    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:15.768719 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:13:15.768669    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79100f5f_7848_488a_819a_eb9227e0e055.slice/crio-4f47ae1dd9d1c11add9ca37681052d7ba8e794f5e3669a07b032bb28b00bc8ab WatchSource:0}: Error finding container 4f47ae1dd9d1c11add9ca37681052d7ba8e794f5e3669a07b032bb28b00bc8ab: Status 404 returned error can't find the container with id 4f47ae1dd9d1c11add9ca37681052d7ba8e794f5e3669a07b032bb28b00bc8ab
Oct 30 13:13:15.769375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.769343    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"]
Oct 30 13:13:15.774628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.774586    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:15.774773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.774680    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:15.774773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:15.774726    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:16.614345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:16.614298    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" event={"ID":"79100f5f-7848-488a-819a-eb9227e0e055","Type":"ContainerStarted","Data":"6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f"}
Oct 30 13:13:16.614869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:16.614416    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" event={"ID":"79100f5f-7848-488a-819a-eb9227e0e055","Type":"ContainerStarted","Data":"4f47ae1dd9d1c11add9ca37681052d7ba8e794f5e3669a07b032bb28b00bc8ab"}
Oct 30 13:13:16.693275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:16.693196    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podStartSLOduration=2.693178031 podStartE2EDuration="2.693178031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:16.679734005 +0000 UTC m=+2461.155453653" watchObservedRunningTime="2026-10-30 13:13:16.693178031 +0000 UTC m=+2461.168897734"
Oct 30 13:13:17.360277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:17.354733    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:17.377917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:17.377880    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:17.624858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:17.624765    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:17.625959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:17.625936    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:13:18.615103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:18.615068    4993 scope.go:122] "RemoveContainer" containerID="d733c7603098e83dcb757ff1b58a646551768e76c4310cf13a39bb4c4b946df9"
Oct 30 13:13:35.805874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.804857    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"]
Oct 30 13:13:35.814260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.814233    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:35.832358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.831418    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"]
Oct 30 13:13:35.838138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.838109    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-e9cb4a\"/\"auth-test-model-e9cb4a-kserve-self-signed-certs\""
Oct 30 13:13:35.838138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.838133    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-e9cb4a\"/\"default-dockercfg-2ktrv\""
Oct 30 13:13:35.848348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.848308    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-e9cb4a\"/\"openshift-service-ca.crt\""
Oct 30 13:13:35.848560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.848417    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-e9cb4a\"/\"kube-root-ca.crt\""
Oct 30 13:13:35.910827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.910783    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-home\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:35.911089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.911063    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-dshm\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:35.911278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.911258    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcn8\" (UniqueName: \"kubernetes.io/projected/4578924e-2c82-4845-a56c-149ac69bfd21-kube-api-access-nmcn8\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:35.911437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.911419    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-model-cache\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:35.911590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.911572    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4578924e-2c82-4845-a56c-149ac69bfd21-tls-certs\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:35.911704 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:35.911691    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-tmp-dir\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.013535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.013488    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-home\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.013535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.013538    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-dshm\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.013845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.013575    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcn8\" (UniqueName: \"kubernetes.io/projected/4578924e-2c82-4845-a56c-149ac69bfd21-kube-api-access-nmcn8\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.013845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.013595    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-model-cache\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.013845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.013620    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4578924e-2c82-4845-a56c-149ac69bfd21-tls-certs\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.013845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.013636    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-tmp-dir\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.014067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.013962    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-home\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.014067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.014009    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-tmp-dir\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.014296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.014271    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-model-cache\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.015995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.015967    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-dshm\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.016498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.016472    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4578924e-2c82-4845-a56c-149ac69bfd21-tls-certs\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.040745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.040704    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcn8\" (UniqueName: \"kubernetes.io/projected/4578924e-2c82-4845-a56c-149ac69bfd21-kube-api-access-nmcn8\") pod \"auth-test-model-e9cb4a-kserve-7c78668469-hnggf\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.140584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.140489    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:36.692906 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:13:36.692854    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4578924e_2c82_4845_a56c_149ac69bfd21.slice/crio-b64c1e9e2699bb03e4ef6d19407064f51286371653a2035e60860b8c03c15e93 WatchSource:0}: Error finding container b64c1e9e2699bb03e4ef6d19407064f51286371653a2035e60860b8c03c15e93: Status 404 returned error can't find the container with id b64c1e9e2699bb03e4ef6d19407064f51286371653a2035e60860b8c03c15e93
Oct 30 13:13:36.695625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.695598    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"]
Oct 30 13:13:36.889234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.881619    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf" event={"ID":"4578924e-2c82-4845-a56c-149ac69bfd21","Type":"ContainerStarted","Data":"25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269"}
Oct 30 13:13:36.889234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.881679    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf" event={"ID":"4578924e-2c82-4845-a56c-149ac69bfd21","Type":"ContainerStarted","Data":"b64c1e9e2699bb03e4ef6d19407064f51286371653a2035e60860b8c03c15e93"}
Oct 30 13:13:36.928991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:36.928926    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf" podStartSLOduration=1.928912966 podStartE2EDuration="1.928912966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:36.927870296 +0000 UTC m=+2481.403589907" watchObservedRunningTime="2026-10-30 13:13:36.928912966 +0000 UTC m=+2481.404632576"
Oct 30 13:13:37.148274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:37.148238    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:38.574593 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:13:38.574547    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:13:41.929653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:41.929613    4993 generic.go:356] "Generic (PLEG): container finished" podID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerID="bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72" exitCode=0
Oct 30 13:13:41.930198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:41.929697    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" event={"ID":"12584623-fe58-48cb-b84c-2dc5012a7a7d","Type":"ContainerDied","Data":"bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72"}
Oct 30 13:13:41.930321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:41.930256    4993 scope.go:122] "RemoveContainer" containerID="bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72"
Oct 30 13:13:41.932118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:41.932086    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:41.932235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:41.932162    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:41.932235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:41.932198    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:42.940900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:42.940869    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" event={"ID":"12584623-fe58-48cb-b84c-2dc5012a7a7d","Type":"ContainerStarted","Data":"63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85"}
Oct 30 13:13:43.046556 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:43.046402    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:13:43.047151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:43.047120    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:13:43.047352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:43.047174    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 13:13:44.044540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:44.044507    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:13:44.044939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:44.044559    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 13:13:45.045198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:45.045161    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:13:45.045643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:45.045229    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 13:13:46.045042 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:46.045004    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:13:46.045267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:46.045071    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 13:13:46.147444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:46.147400    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:46.189856 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:46.189817    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:47.018250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:47.017853    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:13:47.044606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:47.044567    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:13:47.044792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:47.044652    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 13:13:48.044708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:48.044675    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:13:48.045139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:48.044732    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 13:13:49.049286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:49.049254    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:13:49.049790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:49.049769    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:13:49.050427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:13:49.050406    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:14:08.572580 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:14:08.572531    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:14:10.596800 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:10.596759    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"]
Oct 30 13:14:10.599546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:10.598252    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" containerID="cri-o://f1abcef2134913a581188e0daf86c908f15067b01183e84e1cb5ef162a73d234" gracePeriod=30
Oct 30 13:14:12.293547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:12.293509    4993 generic.go:356] "Generic (PLEG): container finished" podID="79100f5f-7848-488a-819a-eb9227e0e055" containerID="6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f" exitCode=0
Oct 30 13:14:12.294114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:12.293588    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" event={"ID":"79100f5f-7848-488a-819a-eb9227e0e055","Type":"ContainerDied","Data":"6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f"}
Oct 30 13:14:12.294189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:12.294138    4993 scope.go:122] "RemoveContainer" containerID="6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f"
Oct 30 13:14:12.296005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:12.295980    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:12.296103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:12.296048    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:12.296103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:12.296079    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:13.303736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:13.303694    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" event={"ID":"79100f5f-7848-488a-819a-eb9227e0e055","Type":"ContainerStarted","Data":"fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b"}
Oct 30 13:14:13.370098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:13.363395    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:14:13.378411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:13.371831    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:14:13.378411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:13.371891    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 13:14:14.373232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:14.370658    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:14:14.373232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:14.370727    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 13:14:15.358135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:15.358098    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:14:15.358482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:15.358437    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 13:14:15.635501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:15.634516    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:14:15.635501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:15.634569    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 13:14:16.342471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.339396    4993 generic.go:356] "Generic (PLEG): container finished" podID="07ba323d-a154-4386-a080-1999960e130c" containerID="f1abcef2134913a581188e0daf86c908f15067b01183e84e1cb5ef162a73d234" exitCode=0
Oct 30 13:14:16.342471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.339476    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" event={"ID":"07ba323d-a154-4386-a080-1999960e130c","Type":"ContainerDied","Data":"f1abcef2134913a581188e0daf86c908f15067b01183e84e1cb5ef162a73d234"}
Oct 30 13:14:16.356237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.354616    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:14:16.356237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.354673    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 13:14:16.404053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.403990    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500294    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ba323d-a154-4386-a080-1999960e130c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba323d-a154-4386-a080-1999960e130c-istiod-ca-cert\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500423    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-socket\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500456    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-kube-api-access-9mtrt\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-kube-api-access-9mtrt\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500503    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-istio-token\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-istio-token\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500548    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-certs\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500598    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-data\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500657    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07ba323d-a154-4386-a080-1999960e130c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba323d-a154-4386-a080-1999960e130c-istio-podinfo\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500839    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-envoy\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.500907    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-credential-socket\") pod \"07ba323d-a154-4386-a080-1999960e130c\" (UID: \"07ba323d-a154-4386-a080-1999960e130c\") "
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.501966    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-credential-socket" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.502314    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-certs" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.502440    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-data" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:16.502897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.502507    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-socket" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:16.504629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.504585    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ba323d-a154-4386-a080-1999960e130c-istiod-ca-cert" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:16.506146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.506115    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07ba323d-a154-4386-a080-1999960e130c-istio-podinfo" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:14:16.512455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.512430    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-envoy" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:16.514949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.514925    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-kube-api-access-9mtrt" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "kube-api-access-9mtrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:16.515396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.515369    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-istio-token" pod "07ba323d-a154-4386-a080-1999960e130c" (UID: "07ba323d-a154-4386-a080-1999960e130c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:16.605349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605309    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:16.605349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605350    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtrt\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-kube-api-access-9mtrt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:16.605618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605365    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ba323d-a154-4386-a080-1999960e130c-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:16.605618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605379    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:16.605618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605392    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:16.605618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605403    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ba323d-a154-4386-a080-1999960e130c-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:16.605618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605416    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:16.605618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605428    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ba323d-a154-4386-a080-1999960e130c-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:16.605618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:16.605444    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ba323d-a154-4386-a080-1999960e130c-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:14:17.349575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.349547    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"
Oct 30 13:14:17.350055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.349592    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8" event={"ID":"07ba323d-a154-4386-a080-1999960e130c","Type":"ContainerDied","Data":"f3e21f8e6c0b363b426f4ce9e37647c67be7a91276f6c9f32c8d18908d93d058"}
Oct 30 13:14:17.350055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.349666    4993 scope.go:122] "RemoveContainer" containerID="f1abcef2134913a581188e0daf86c908f15067b01183e84e1cb5ef162a73d234"
Oct 30 13:14:17.357018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.356984    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:14:17.357314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.357260    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 13:14:17.495259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.485154    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"]
Oct 30 13:14:17.501641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.501606    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-j4ts8"]
Oct 30 13:14:17.625436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.625343    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:14:17.694444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:17.694407    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ba323d-a154-4386-a080-1999960e130c" path="/var/lib/kubelet/pods/07ba323d-a154-4386-a080-1999960e130c/volumes"
Oct 30 13:14:18.357663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:18.357614    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:14:18.358241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:18.357700    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 13:14:18.655955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:18.655862    4993 scope.go:122] "RemoveContainer" containerID="1f9214caa07893c72c6894bedfc41e50ab0d27c5350164724ef4aff44db98ab8"
Oct 30 13:14:19.359906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.359873    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:14:19.360978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.360951    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:14:19.710893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.710771    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"]
Oct 30 13:14:19.711661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.711634    4993 cpu_manager.go:414] "containerMap: removing container" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy"
Oct 30 13:14:19.711661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.711660    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy"
Oct 30 13:14:19.711876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.711845    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ba323d-a154-4386-a080-1999960e130c" containerName="istio-proxy"
Oct 30 13:14:19.716830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.716802    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.738291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.738220    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"]
Oct 30 13:14:19.742428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.742391    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-92855c\"/\"openshift-service-ca.crt\""
Oct 30 13:14:19.742607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.742451    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-92855c\"/\"kube-root-ca.crt\""
Oct 30 13:14:19.742607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.742527    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-92855c\"/\"default-dockercfg-42w2p\""
Oct 30 13:14:19.742742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.742662    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-92855c\"/\"auth-test-model-92855c-kserve-self-signed-certs\""
Oct 30 13:14:19.755997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.755955    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2838ebe2-712b-4d88-881d-c4c17c6308f6-tls-certs\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.756282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.756031    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-home\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.756282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.756200    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-model-cache\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.756487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.756305    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-tmp-dir\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.756487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.756335    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l489f\" (UniqueName: \"kubernetes.io/projected/2838ebe2-712b-4d88-881d-c4c17c6308f6-kube-api-access-l489f\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.756487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.756416    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-dshm\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.857929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.857894    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2838ebe2-712b-4d88-881d-c4c17c6308f6-tls-certs\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.858166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.857959    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-home\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.858166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.858089    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-model-cache\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.858166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.858145    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-tmp-dir\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.858375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.858170    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l489f\" (UniqueName: \"kubernetes.io/projected/2838ebe2-712b-4d88-881d-c4c17c6308f6-kube-api-access-l489f\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.858375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.858287    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-dshm\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.858489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.858466    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-home\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.858546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.858521    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-model-cache\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.858835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.858807    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-tmp-dir\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.861248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.861182    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-dshm\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.861613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.861590    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2838ebe2-712b-4d88-881d-c4c17c6308f6-tls-certs\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:19.884163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:19.884127    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l489f\" (UniqueName: \"kubernetes.io/projected/2838ebe2-712b-4d88-881d-c4c17c6308f6-kube-api-access-l489f\") pod \"auth-test-model-92855c-kserve-557cc7f749-txlhs\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") " pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:20.039548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:20.039505    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:20.445304 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:14:20.445261    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2838ebe2_712b_4d88_881d_c4c17c6308f6.slice/crio-b4d16667027f87b080ee2c7844078ce04eab4bff3b4f60d1dbf2c80fdc9eb9f1 WatchSource:0}: Error finding container b4d16667027f87b080ee2c7844078ce04eab4bff3b4f60d1dbf2c80fdc9eb9f1: Status 404 returned error can't find the container with id b4d16667027f87b080ee2c7844078ce04eab4bff3b4f60d1dbf2c80fdc9eb9f1
Oct 30 13:14:20.446624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:20.446598    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"]
Oct 30 13:14:21.415999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:21.415960    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs" event={"ID":"2838ebe2-712b-4d88-881d-c4c17c6308f6","Type":"ContainerStarted","Data":"6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4"}
Oct 30 13:14:21.416239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:21.416011    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs" event={"ID":"2838ebe2-712b-4d88-881d-c4c17c6308f6","Type":"ContainerStarted","Data":"b4d16667027f87b080ee2c7844078ce04eab4bff3b4f60d1dbf2c80fdc9eb9f1"}
Oct 30 13:14:21.465290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:21.465195    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs" podStartSLOduration=2.465181862 podStartE2EDuration="2.465181862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:21.461752442 +0000 UTC m=+2525.937472051" watchObservedRunningTime="2026-10-30 13:14:21.465181862 +0000 UTC m=+2525.940901473"
Oct 30 13:14:22.041286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:22.041247    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:28.045647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:28.045607    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"]
Oct 30 13:14:28.061989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:28.061938    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-556b7370-wpd88"]
Oct 30 13:14:29.694528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:29.694489    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21397549-3a57-492e-9efe-8cfc9664fbb9" path="/var/lib/kubelet/pods/21397549-3a57-492e-9efe-8cfc9664fbb9/volumes"
Oct 30 13:14:30.041431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.041386    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:30.056470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.056430    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:30.562251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.562199    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:14:30.789545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.789503    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"]
Oct 30 13:14:30.793646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.793618    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.818819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.818732    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"]
Oct 30 13:14:30.818819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.818787    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-bd2aaf\"/\"kube-root-ca.crt\""
Oct 30 13:14:30.819059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.818895    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-bd2aaf\"/\"openshift-service-ca.crt\""
Oct 30 13:14:30.819059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.818942    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-bd2aaf\"/\"default-dockercfg-858h6\""
Oct 30 13:14:30.819059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.818996    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-bd2aaf\"/\"auth-test-model-bd2aaf-kserve-self-signed-certs\""
Oct 30 13:14:30.857417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.857320    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-home\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.857719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.857686    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-model-cache\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.857843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.857745    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b9f157-a886-415e-b6f1-dfa1364462d3-tls-certs\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.857843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.857797    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mztdb\" (UniqueName: \"kubernetes.io/projected/97b9f157-a886-415e-b6f1-dfa1364462d3-kube-api-access-mztdb\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.858032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.858010    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-dshm\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.858397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.858260    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-tmp-dir\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.959733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.959680    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-tmp-dir\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.959969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.959848    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-home\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.959969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.959942    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-model-cache\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.960251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.960226    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-tmp-dir\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.960331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.960276    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-home\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.960397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.960368    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-model-cache\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.960397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.960383    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b9f157-a886-415e-b6f1-dfa1364462d3-tls-certs\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.960515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.960461    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mztdb\" (UniqueName: \"kubernetes.io/projected/97b9f157-a886-415e-b6f1-dfa1364462d3-kube-api-access-mztdb\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.960515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.960502    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-dshm\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.963933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.963891    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b9f157-a886-415e-b6f1-dfa1364462d3-tls-certs\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.964495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.964469    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-dshm\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:30.986544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:30.986505    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mztdb\" (UniqueName: \"kubernetes.io/projected/97b9f157-a886-415e-b6f1-dfa1364462d3-kube-api-access-mztdb\") pod \"auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") " pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:31.107473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:31.107365    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:31.595107 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:14:31.595073    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97b9f157_a886_415e_b6f1_dfa1364462d3.slice/crio-44d04176d349565e5b7aef8a8cb001a69d476135bc9fdb623020a8931fe73840 WatchSource:0}: Error finding container 44d04176d349565e5b7aef8a8cb001a69d476135bc9fdb623020a8931fe73840: Status 404 returned error can't find the container with id 44d04176d349565e5b7aef8a8cb001a69d476135bc9fdb623020a8931fe73840
Oct 30 13:14:31.599164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:31.599124    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"]
Oct 30 13:14:32.574132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:32.574090    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf" event={"ID":"97b9f157-a886-415e-b6f1-dfa1364462d3","Type":"ContainerStarted","Data":"165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b"}
Oct 30 13:14:32.574638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:32.574144    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf" event={"ID":"97b9f157-a886-415e-b6f1-dfa1364462d3","Type":"ContainerStarted","Data":"44d04176d349565e5b7aef8a8cb001a69d476135bc9fdb623020a8931fe73840"}
Oct 30 13:14:32.626518 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:32.626454    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf" podStartSLOduration=2.626437641 podStartE2EDuration="2.626437641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:32.623986952 +0000 UTC m=+2537.099706561" watchObservedRunningTime="2026-10-30 13:14:32.626437641 +0000 UTC m=+2537.102157251"
Oct 30 13:14:33.107704 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:33.107658    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:38.574926 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:14:38.574878    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:14:41.108278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:41.108199    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:41.122597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:41.122565    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:41.757769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:41.757721    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:14:43.032946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:43.032906    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"]
Oct 30 13:14:43.046824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:43.046791    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7b4490c-28rf9"]
Oct 30 13:14:43.694081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:43.694037    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd737ed-3c76-458d-afb1-7aba8e386892" path="/var/lib/kubelet/pods/8cd737ed-3c76-458d-afb1-7aba8e386892/volumes"
Oct 30 13:14:45.035890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:45.035853    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"]
Oct 30 13:14:45.049450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:45.049410    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4ae85f-g9j4t"]
Oct 30 13:14:45.694621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:45.694584    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25607eb-52ce-4e2d-a176-e5f42c1012ae" path="/var/lib/kubelet/pods/a25607eb-52ce-4e2d-a176-e5f42c1012ae/volumes"
Oct 30 13:14:55.195764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:55.195721    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"]
Oct 30 13:14:55.196282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:55.195940    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy" containerID="cri-o://60282eac81f101656848f91e10a17c22db2969fd53bd6f5385f7d098fd706ca7" gracePeriod=30
Oct 30 13:14:57.038404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:57.038360    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"]
Oct 30 13:14:57.052402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:57.052348    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949d6391-xvtfd"]
Oct 30 13:14:57.693477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:57.693435    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3490bc-b617-4cb0-a765-efe4d384647f" path="/var/lib/kubelet/pods/ba3490bc-b617-4cb0-a765-efe4d384647f/volumes"
Oct 30 13:14:57.966940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:57.966854    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mzhxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:14:57.966940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:57.966931    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:14:58.043268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:58.043222    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"]
Oct 30 13:14:58.056850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:58.056809    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11fc584-b7r7c"]
Oct 30 13:14:59.035722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:59.035684    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"]
Oct 30 13:14:59.050271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:59.050229    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b851af-mv8h9"]
Oct 30 13:14:59.694254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:59.694198    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6" path="/var/lib/kubelet/pods/1ee7dea9-5e02-4a05-aa98-ee0cb578e0a6/volumes"
Oct 30 13:14:59.694727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:14:59.694712    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8" path="/var/lib/kubelet/pods/fd6eaa5d-1c8d-4231-9a54-fc43b9afcec8/volumes"
Oct 30 13:15:00.159951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.159913    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"]
Oct 30 13:15:00.164253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.164223    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:00.164638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.164606    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"]
Oct 30 13:15:00.168855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.168795    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:00.170261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.170234    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"]
Oct 30 13:15:00.175288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.175257    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:00.176350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.176324    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"]
Oct 30 13:15:00.180763 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.180736    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:00.182107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.182081    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"]
Oct 30 13:15:00.186070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.186043    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:00.194493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.194450    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:15:00.198641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.198607    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:15:00.201148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.201121    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:15:00.215587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.215545    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"]
Oct 30 13:15:00.220003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.219967    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"]
Oct 30 13:15:00.232923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.230633    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"]
Oct 30 13:15:00.237425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.237382    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"]
Oct 30 13:15:00.241373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.241349    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:00.242914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.242576    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"]
Oct 30 13:15:00.248846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.248718    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"]
Oct 30 13:15:00.276262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.273195    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"]
Oct 30 13:15:00.276469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.276416    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:00.277694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.277665    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:00.277811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.277573    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"]
Oct 30 13:15:00.286272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.282143    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:00.286272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.284973    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"]
Oct 30 13:15:00.290024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.289998    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"]
Oct 30 13:15:00.328717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.328684    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjnt\" (UniqueName: \"kubernetes.io/projected/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-kube-api-access-9wjnt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx\" (UID: \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:00.328919 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.328903    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976e31a-dc21-49b8-b7b1-31084a53d7e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq\" (UID: \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:00.329026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.329015    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w\" (UID: \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:00.329129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.329116    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlsr4\" (UniqueName: \"kubernetes.io/projected/02ecd841-a5d5-43db-bb17-4b05e897b59b-kube-api-access-rlsr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g\" (UID: \"02ecd841-a5d5-43db-bb17-4b05e897b59b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:00.334898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.334367    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ecd841-a5d5-43db-bb17-4b05e897b59b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g\" (UID: \"02ecd841-a5d5-43db-bb17-4b05e897b59b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:00.334898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.334425    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596d4\" (UniqueName: \"kubernetes.io/projected/2976e31a-dc21-49b8-b7b1-31084a53d7e3-kube-api-access-596d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq\" (UID: \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:00.334898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.334506    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s\" (UID: \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:00.334898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.334573    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx\" (UID: \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:00.334898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.334643    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c9hl\" (UniqueName: \"kubernetes.io/projected/dc13c75b-d11f-4444-af69-5db6694deea3-kube-api-access-5c9hl\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d\" (UID: \"dc13c75b-d11f-4444-af69-5db6694deea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:00.334898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.334748    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k64tv\" (UniqueName: \"kubernetes.io/projected/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-kube-api-access-k64tv\") pod \"maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s\" (UID: \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:00.335515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.334919    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzfph\" (UniqueName: \"kubernetes.io/projected/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-kube-api-access-kzfph\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w\" (UID: \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:00.335515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.334981    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc13c75b-d11f-4444-af69-5db6694deea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d\" (UID: \"dc13c75b-d11f-4444-af69-5db6694deea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:00.337981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.337455    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"]
Oct 30 13:15:00.350277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.350245    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"]
Oct 30 13:15:00.376028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.375988    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"]
Oct 30 13:15:00.376375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.376350    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:00.383090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.383054    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"]
Oct 30 13:15:00.383273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.383104    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"]
Oct 30 13:15:00.383273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.383132    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"]
Oct 30 13:15:00.383273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.383247    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:00.394475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.394442    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"]
Oct 30 13:15:00.436032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.435930    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p\" (UID: \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:00.436032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.435988    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrxt\" (UniqueName: \"kubernetes.io/projected/d3893102-450d-4e3d-974f-f57d67b662de-kube-api-access-xlrxt\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6\" (UID: \"d3893102-450d-4e3d-974f-f57d67b662de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:00.436253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436088    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3893102-450d-4e3d-974f-f57d67b662de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6\" (UID: \"d3893102-450d-4e3d-974f-f57d67b662de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:00.436253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436118    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w\" (UID: \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:00.436253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436142    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlsr4\" (UniqueName: \"kubernetes.io/projected/02ecd841-a5d5-43db-bb17-4b05e897b59b-kube-api-access-rlsr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g\" (UID: \"02ecd841-a5d5-43db-bb17-4b05e897b59b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:00.436253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436224    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s\" (UID: \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:00.436479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436260    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbxx\" (UniqueName: \"kubernetes.io/projected/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-kube-api-access-vvbxx\") pod \"maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p\" (UID: \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:00.436479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436292    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dcbb92-946a-4c0f-9627-9acc70f32934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm\" (UID: \"74dcbb92-946a-4c0f-9627-9acc70f32934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:00.436479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436415    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k64tv\" (UniqueName: \"kubernetes.io/projected/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-kube-api-access-k64tv\") pod \"maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s\" (UID: \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:00.436652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436521    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqmt\" (UniqueName: \"kubernetes.io/projected/74dcbb92-946a-4c0f-9627-9acc70f32934-kube-api-access-qfqmt\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm\" (UID: \"74dcbb92-946a-4c0f-9627-9acc70f32934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:00.436652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436579    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzfph\" (UniqueName: \"kubernetes.io/projected/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-kube-api-access-kzfph\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w\" (UID: \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:00.436652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436621    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc13c75b-d11f-4444-af69-5db6694deea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d\" (UID: \"dc13c75b-d11f-4444-af69-5db6694deea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:00.436825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436675    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjnt\" (UniqueName: \"kubernetes.io/projected/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-kube-api-access-9wjnt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx\" (UID: \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:00.436825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436701    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976e31a-dc21-49b8-b7b1-31084a53d7e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq\" (UID: \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:00.436825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436761    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ecd841-a5d5-43db-bb17-4b05e897b59b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g\" (UID: \"02ecd841-a5d5-43db-bb17-4b05e897b59b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:00.436825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436812    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596d4\" (UniqueName: \"kubernetes.io/projected/2976e31a-dc21-49b8-b7b1-31084a53d7e3-kube-api-access-596d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq\" (UID: \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:00.437046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436879    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx\" (UID: \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:00.437046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436922    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c9hl\" (UniqueName: \"kubernetes.io/projected/dc13c75b-d11f-4444-af69-5db6694deea3-kube-api-access-5c9hl\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d\" (UID: \"dc13c75b-d11f-4444-af69-5db6694deea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:00.437046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.436978    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w\" (UID: \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:00.437188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.437096    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s\" (UID: \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:00.437496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.437470    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc13c75b-d11f-4444-af69-5db6694deea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d\" (UID: \"dc13c75b-d11f-4444-af69-5db6694deea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:00.437566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.437541    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976e31a-dc21-49b8-b7b1-31084a53d7e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq\" (UID: \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:00.437646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.437591    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ecd841-a5d5-43db-bb17-4b05e897b59b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g\" (UID: \"02ecd841-a5d5-43db-bb17-4b05e897b59b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:00.437778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.437756    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx\" (UID: \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:00.468944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.468876    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzfph\" (UniqueName: \"kubernetes.io/projected/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-kube-api-access-kzfph\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w\" (UID: \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:00.469152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.469057    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjnt\" (UniqueName: \"kubernetes.io/projected/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-kube-api-access-9wjnt\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx\" (UID: \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:00.470760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.470729    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c9hl\" (UniqueName: \"kubernetes.io/projected/dc13c75b-d11f-4444-af69-5db6694deea3-kube-api-access-5c9hl\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d\" (UID: \"dc13c75b-d11f-4444-af69-5db6694deea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:00.470760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.470744    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k64tv\" (UniqueName: \"kubernetes.io/projected/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-kube-api-access-k64tv\") pod \"maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s\" (UID: \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:00.471449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.471423    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596d4\" (UniqueName: \"kubernetes.io/projected/2976e31a-dc21-49b8-b7b1-31084a53d7e3-kube-api-access-596d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq\" (UID: \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:00.471877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.471853    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlsr4\" (UniqueName: \"kubernetes.io/projected/02ecd841-a5d5-43db-bb17-4b05e897b59b-kube-api-access-rlsr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g\" (UID: \"02ecd841-a5d5-43db-bb17-4b05e897b59b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:00.485148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.485113    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:00.511651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.511618    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:00.538817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.538782    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p\" (UID: \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:00.539004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.538835    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrxt\" (UniqueName: \"kubernetes.io/projected/d3893102-450d-4e3d-974f-f57d67b662de-kube-api-access-xlrxt\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6\" (UID: \"d3893102-450d-4e3d-974f-f57d67b662de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:00.539004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.538904    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3893102-450d-4e3d-974f-f57d67b662de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6\" (UID: \"d3893102-450d-4e3d-974f-f57d67b662de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:00.539004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.538969    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbxx\" (UniqueName: \"kubernetes.io/projected/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-kube-api-access-vvbxx\") pod \"maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p\" (UID: \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:00.539004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.538996    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dcbb92-946a-4c0f-9627-9acc70f32934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm\" (UID: \"74dcbb92-946a-4c0f-9627-9acc70f32934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:00.539248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.539092    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqmt\" (UniqueName: \"kubernetes.io/projected/74dcbb92-946a-4c0f-9627-9acc70f32934-kube-api-access-qfqmt\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm\" (UID: \"74dcbb92-946a-4c0f-9627-9acc70f32934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:00.539248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.539188    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh\" (UID: \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:00.539364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.539304    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt\" (UID: \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:00.539364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.539335    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54khz\" (UniqueName: \"kubernetes.io/projected/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-kube-api-access-54khz\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt\" (UID: \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:00.539486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.539372    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rsn\" (UniqueName: \"kubernetes.io/projected/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-kube-api-access-v8rsn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh\" (UID: \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:00.539486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.539462    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p\" (UID: \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:00.540283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.540256    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3893102-450d-4e3d-974f-f57d67b662de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6\" (UID: \"d3893102-450d-4e3d-974f-f57d67b662de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:00.540466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.540413    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dcbb92-946a-4c0f-9627-9acc70f32934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm\" (UID: \"74dcbb92-946a-4c0f-9627-9acc70f32934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:00.580622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.579932    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrxt\" (UniqueName: \"kubernetes.io/projected/d3893102-450d-4e3d-974f-f57d67b662de-kube-api-access-xlrxt\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6\" (UID: \"d3893102-450d-4e3d-974f-f57d67b662de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:00.581032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.580964    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqmt\" (UniqueName: \"kubernetes.io/projected/74dcbb92-946a-4c0f-9627-9acc70f32934-kube-api-access-qfqmt\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm\" (UID: \"74dcbb92-946a-4c0f-9627-9acc70f32934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:00.590175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.583694    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbxx\" (UniqueName: \"kubernetes.io/projected/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-kube-api-access-vvbxx\") pod \"maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p\" (UID: \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:00.617232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.617180    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:00.633546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.633513    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:00.643895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.642241    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh\" (UID: \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:00.643895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.642359    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt\" (UID: \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:00.643895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.642390    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54khz\" (UniqueName: \"kubernetes.io/projected/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-kube-api-access-54khz\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt\" (UID: \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:00.643895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.642421    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rsn\" (UniqueName: \"kubernetes.io/projected/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-kube-api-access-v8rsn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh\" (UID: \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:00.643895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.642949    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh\" (UID: \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:00.645991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.645524    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt\" (UID: \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:00.645991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.645879    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:00.646286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.646238    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:00.657902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.657869    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:00.677886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.676082    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:00.691689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.691612    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:00.695729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.695657    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rsn\" (UniqueName: \"kubernetes.io/projected/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-kube-api-access-v8rsn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh\" (UID: \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:00.704987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.704414    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:00.720730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.720698    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54khz\" (UniqueName: \"kubernetes.io/projected/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-kube-api-access-54khz\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt\" (UID: \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:00.835529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.835483    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"]
Oct 30 13:15:00.906873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.905546    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"]
Oct 30 13:15:00.951639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.951601    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" event={"ID":"dc13c75b-d11f-4444-af69-5db6694deea3","Type":"ContainerStarted","Data":"7788f6fa95963a846f00e5f7fee7869e0515c60bdedc53a86215028fde1890b3"}
Oct 30 13:15:00.957308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.956631    4993 generic.go:356] "Generic (PLEG): container finished" podID="1a788667-becf-429a-af19-7df65728c6bd" containerID="60282eac81f101656848f91e10a17c22db2969fd53bd6f5385f7d098fd706ca7" exitCode=0
Oct 30 13:15:00.957308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.956840    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" event={"ID":"1a788667-becf-429a-af19-7df65728c6bd","Type":"ContainerDied","Data":"60282eac81f101656848f91e10a17c22db2969fd53bd6f5385f7d098fd706ca7"}
Oct 30 13:15:00.958954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:00.958882    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" event={"ID":"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a","Type":"ContainerStarted","Data":"85f85358721fdb879d6bf37e27e97af6e00d6c5357ad4598aab457a2755f7c13"}
Oct 30 13:15:01.007806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.007737    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"]
Oct 30 13:15:01.009860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.009829    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:01.056966 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.056934    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"]
Oct 30 13:15:01.096338 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:15:01.096303    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ecd841_a5d5_43db_bb17_4b05e897b59b.slice/crio-9dadb46a0075a9bea179352f44d578b8492921ec41b92aaa23e5d38644839571 WatchSource:0}: Error finding container 9dadb46a0075a9bea179352f44d578b8492921ec41b92aaa23e5d38644839571: Status 404 returned error can't find the container with id 9dadb46a0075a9bea179352f44d578b8492921ec41b92aaa23e5d38644839571
Oct 30 13:15:01.178224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.178187    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 13:15:01.343009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.342868    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-socket\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.344432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.343299    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-envoy\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.344432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.343363    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-data\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.344432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.343438    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a788667-becf-429a-af19-7df65728c6bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a788667-becf-429a-af19-7df65728c6bd-istio-podinfo\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.344432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.343528    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a788667-becf-429a-af19-7df65728c6bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a788667-becf-429a-af19-7df65728c6bd-istiod-ca-cert\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.344432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.343571    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-kube-api-access-qgg6b\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-kube-api-access-qgg6b\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.344432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.343610    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-certs\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.344432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.343699    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-istio-token\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-istio-token\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.345243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.344951    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-credential-socket\") pod \"1a788667-becf-429a-af19-7df65728c6bd\" (UID: \"1a788667-becf-429a-af19-7df65728c6bd\") "
Oct 30 13:15:01.350246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.349686    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a788667-becf-429a-af19-7df65728c6bd-istiod-ca-cert" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:01.350400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.350378    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a788667-becf-429a-af19-7df65728c6bd-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.350837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.350536    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-credential-socket" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:01.350837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.350550    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-data" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:01.350837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.350687    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-socket" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:01.350837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.350752    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-certs" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:01.374156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.374100    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-envoy" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:01.375285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.375198    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-istio-token" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:01.377030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.376992    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-kube-api-access-qgg6b" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "kube-api-access-qgg6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:01.385530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.379875    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a788667-becf-429a-af19-7df65728c6bd-istio-podinfo" pod "1a788667-becf-429a-af19-7df65728c6bd" (UID: "1a788667-becf-429a-af19-7df65728c6bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:01.424834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.424795    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"]
Oct 30 13:15:01.443659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.443632    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"]
Oct 30 13:15:01.448634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.448606    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"]
Oct 30 13:15:01.460462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.457109    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.460462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.457135    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.460462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.457150    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.460462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.457163    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a788667-becf-429a-af19-7df65728c6bd-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.460462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.457175    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgg6b\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-kube-api-access-qgg6b\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.460462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.457191    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.460462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.457217    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a788667-becf-429a-af19-7df65728c6bd-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.460462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.457230    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a788667-becf-429a-af19-7df65728c6bd-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.490756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.484318    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"]
Oct 30 13:15:01.490958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.490931    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"]
Oct 30 13:15:01.557541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.557501    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"]
Oct 30 13:15:01.642326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.642291    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"]
Oct 30 13:15:01.973979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.973937    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" event={"ID":"dc13c75b-d11f-4444-af69-5db6694deea3","Type":"ContainerStarted","Data":"86c03c72d751bf334eb3f740da9b93d918aad357de836087a793a07a7be39fcf"}
Oct 30 13:15:01.975729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.975706    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"
Oct 30 13:15:01.975877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.975736    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg" event={"ID":"1a788667-becf-429a-af19-7df65728c6bd","Type":"ContainerDied","Data":"0c7561bed85e3e615d0ddd800cca82979a4acc5240de1389aaf9abba87cd2021"}
Oct 30 13:15:01.975877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.975795    4993 scope.go:122] "RemoveContainer" containerID="60282eac81f101656848f91e10a17c22db2969fd53bd6f5385f7d098fd706ca7"
Oct 30 13:15:01.980110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.980083    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" event={"ID":"50a59e6b-4b6c-4bf4-b5d6-86462b68e611","Type":"ContainerStarted","Data":"35444618a08f1dcad53a57f553b16b897057b6118c83f151b80d67813329f180"}
Oct 30 13:15:01.989163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.989100    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" event={"ID":"bb18ce19-72f1-42ad-8301-b8c5903f9aaa","Type":"ContainerStarted","Data":"2a993d9393aa755ffa931ba1d029f16a6ed8b92906feef01b631b2c10cb5d58b"}
Oct 30 13:15:01.991614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.991582    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" event={"ID":"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759","Type":"ContainerStarted","Data":"93879694d9014dafac04d1fac8badc8cfecdf6ff64ea249392427b04da0f40c4"}
Oct 30 13:15:01.994902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.994879    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" event={"ID":"f1116de9-a0c9-4e5e-974f-ac6f8734f32d","Type":"ContainerStarted","Data":"82ac216994cccbb957d883375ccbb64d570e5154ec7f6a556546a972c18ac03a"}
Oct 30 13:15:01.997606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.997118    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" event={"ID":"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a","Type":"ContainerStarted","Data":"22354fae9915c562d918048a3f6c22cf08cd7ac1a8608cb11054814473d0f888"}
Oct 30 13:15:01.999242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.999218    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" event={"ID":"2976e31a-dc21-49b8-b7b1-31084a53d7e3","Type":"ContainerStarted","Data":"90a6be2ad0022f528295f3f89592fce8a203f3753289f82e0baf7ad83031be82"}
Oct 30 13:15:01.999359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:01.999258    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" event={"ID":"2976e31a-dc21-49b8-b7b1-31084a53d7e3","Type":"ContainerStarted","Data":"5cf77089bbe87542e4a53cd8049dbd7ac87825f0bd78c86bbacedf3b5e637202"}
Oct 30 13:15:02.001017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.000995    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" event={"ID":"74dcbb92-946a-4c0f-9627-9acc70f32934","Type":"ContainerStarted","Data":"b45c74780370a6ba0fa0ec89b2b3537519523d3a6173be73ad8d37d077402c2b"}
Oct 30 13:15:02.002542 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.002510    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" event={"ID":"d3893102-450d-4e3d-974f-f57d67b662de","Type":"ContainerStarted","Data":"86bddfb144fbc642c9dd61fae972ea7fd83855f63c67fcc7e2eb965da7004aba"}
Oct 30 13:15:02.003779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.003758    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" event={"ID":"5ce95e7b-befb-4079-a2ab-b83cfd7f741d","Type":"ContainerStarted","Data":"9d9b0a906f28a08b757eca9d3be0ed332ab141807b3d0100d24fde7cd81f07f6"}
Oct 30 13:15:02.005601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.005579    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" event={"ID":"02ecd841-a5d5-43db-bb17-4b05e897b59b","Type":"ContainerStarted","Data":"db916b486b7338ed1893515212c3294dc2e37f9cce140bc336049ca7da5baebb"}
Oct 30 13:15:02.005682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.005613    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" event={"ID":"02ecd841-a5d5-43db-bb17-4b05e897b59b","Type":"ContainerStarted","Data":"9dadb46a0075a9bea179352f44d578b8492921ec41b92aaa23e5d38644839571"}
Oct 30 13:15:02.027458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.027393    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" podStartSLOduration=2.027377526 podStartE2EDuration="2.027377526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:02.022843514 +0000 UTC m=+2566.498563127" watchObservedRunningTime="2026-10-30 13:15:02.027377526 +0000 UTC m=+2566.503097173"
Oct 30 13:15:02.076726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.076664    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" podStartSLOduration=2.076639762 podStartE2EDuration="2.076639762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:02.070728066 +0000 UTC m=+2566.546447677" watchObservedRunningTime="2026-10-30 13:15:02.076639762 +0000 UTC m=+2566.552359377"
Oct 30 13:15:02.116097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.116034    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" podStartSLOduration=2.116017049 podStartE2EDuration="2.116017049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:02.114891772 +0000 UTC m=+2566.590611382" watchObservedRunningTime="2026-10-30 13:15:02.116017049 +0000 UTC m=+2566.591736660"
Oct 30 13:15:02.166128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.166092    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"]
Oct 30 13:15:02.173897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.173867    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mzhxg"]
Oct 30 13:15:02.213662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:02.213602    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" podStartSLOduration=2.213583776 podStartE2EDuration="2.213583776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:02.209419847 +0000 UTC m=+2566.685139457" watchObservedRunningTime="2026-10-30 13:15:02.213583776 +0000 UTC m=+2566.689303387"
Oct 30 13:15:03.022339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.022299    4993 generic.go:356] "Generic (PLEG): container finished" podID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerID="ebfd7500c8423e624fd26b8173eb9c2af59b9701209d4551cfb11bfe318cdb75" exitCode=60
Oct 30 13:15:03.022856 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.022818    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" event={"ID":"bb18ce19-72f1-42ad-8301-b8c5903f9aaa","Type":"ContainerDied","Data":"ebfd7500c8423e624fd26b8173eb9c2af59b9701209d4551cfb11bfe318cdb75"}
Oct 30 13:15:03.023119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.023096    4993 scope.go:122] "RemoveContainer" containerID="ebfd7500c8423e624fd26b8173eb9c2af59b9701209d4551cfb11bfe318cdb75"
Oct 30 13:15:03.025184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.025157    4993 generic.go:356] "Generic (PLEG): container finished" podID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerID="0a0b0abd6971bb8d25b3586ad98737adb2da3e863586059507820196c609a70e" exitCode=60
Oct 30 13:15:03.025463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.025263    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" event={"ID":"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759","Type":"ContainerDied","Data":"0a0b0abd6971bb8d25b3586ad98737adb2da3e863586059507820196c609a70e"}
Oct 30 13:15:03.025593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.025569    4993 scope.go:122] "RemoveContainer" containerID="0a0b0abd6971bb8d25b3586ad98737adb2da3e863586059507820196c609a70e"
Oct 30 13:15:03.028977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.028947    4993 generic.go:356] "Generic (PLEG): container finished" podID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerID="a732492a91f4b4eae3432eaddc327f6600fa8ef9a4d03cf4dbe18b82f379f45d" exitCode=60
Oct 30 13:15:03.029090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.029049    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" event={"ID":"f1116de9-a0c9-4e5e-974f-ac6f8734f32d","Type":"ContainerDied","Data":"a732492a91f4b4eae3432eaddc327f6600fa8ef9a4d03cf4dbe18b82f379f45d"}
Oct 30 13:15:03.029282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.029264    4993 scope.go:122] "RemoveContainer" containerID="a732492a91f4b4eae3432eaddc327f6600fa8ef9a4d03cf4dbe18b82f379f45d"
Oct 30 13:15:03.030844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.030822    4993 generic.go:356] "Generic (PLEG): container finished" podID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerID="22354fae9915c562d918048a3f6c22cf08cd7ac1a8608cb11054814473d0f888" exitCode=60
Oct 30 13:15:03.030929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.030901    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" event={"ID":"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a","Type":"ContainerDied","Data":"22354fae9915c562d918048a3f6c22cf08cd7ac1a8608cb11054814473d0f888"}
Oct 30 13:15:03.031067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.031052    4993 scope.go:122] "RemoveContainer" containerID="22354fae9915c562d918048a3f6c22cf08cd7ac1a8608cb11054814473d0f888"
Oct 30 13:15:03.032715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.032696    4993 generic.go:356] "Generic (PLEG): container finished" podID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerID="90a6be2ad0022f528295f3f89592fce8a203f3753289f82e0baf7ad83031be82" exitCode=60
Oct 30 13:15:03.032796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.032773    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" event={"ID":"2976e31a-dc21-49b8-b7b1-31084a53d7e3","Type":"ContainerDied","Data":"90a6be2ad0022f528295f3f89592fce8a203f3753289f82e0baf7ad83031be82"}
Oct 30 13:15:03.032954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.032935    4993 scope.go:122] "RemoveContainer" containerID="90a6be2ad0022f528295f3f89592fce8a203f3753289f82e0baf7ad83031be82"
Oct 30 13:15:03.034585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.034562    4993 generic.go:356] "Generic (PLEG): container finished" podID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerID="b64f62543708649f6d9c38eae1d7e9720d74805c4dee424856c6ee6dab8ff122" exitCode=60
Oct 30 13:15:03.034703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.034642    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" event={"ID":"74dcbb92-946a-4c0f-9627-9acc70f32934","Type":"ContainerDied","Data":"b64f62543708649f6d9c38eae1d7e9720d74805c4dee424856c6ee6dab8ff122"}
Oct 30 13:15:03.034861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.034844    4993 scope.go:122] "RemoveContainer" containerID="b64f62543708649f6d9c38eae1d7e9720d74805c4dee424856c6ee6dab8ff122"
Oct 30 13:15:03.036525 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.036503    4993 generic.go:356] "Generic (PLEG): container finished" podID="d3893102-450d-4e3d-974f-f57d67b662de" containerID="11ea79146c830cb977f41550b0f6eccbb04b7829db01e5bb841a5eaca2bcd64b" exitCode=60
Oct 30 13:15:03.036998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.036981    4993 scope.go:122] "RemoveContainer" containerID="11ea79146c830cb977f41550b0f6eccbb04b7829db01e5bb841a5eaca2bcd64b"
Oct 30 13:15:03.037248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.037193    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" event={"ID":"d3893102-450d-4e3d-974f-f57d67b662de","Type":"ContainerDied","Data":"11ea79146c830cb977f41550b0f6eccbb04b7829db01e5bb841a5eaca2bcd64b"}
Oct 30 13:15:03.039980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.039957    4993 generic.go:356] "Generic (PLEG): container finished" podID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerID="6ee8a546b7af2e23366c7fb69e1ff844f3e72a648e4bce40de46d511b318e544" exitCode=60
Oct 30 13:15:03.040100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.040035    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" event={"ID":"5ce95e7b-befb-4079-a2ab-b83cfd7f741d","Type":"ContainerDied","Data":"6ee8a546b7af2e23366c7fb69e1ff844f3e72a648e4bce40de46d511b318e544"}
Oct 30 13:15:03.040291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.040270    4993 scope.go:122] "RemoveContainer" containerID="6ee8a546b7af2e23366c7fb69e1ff844f3e72a648e4bce40de46d511b318e544"
Oct 30 13:15:03.042305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.042282    4993 generic.go:356] "Generic (PLEG): container finished" podID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerID="db916b486b7338ed1893515212c3294dc2e37f9cce140bc336049ca7da5baebb" exitCode=60
Oct 30 13:15:03.042553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.042351    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" event={"ID":"02ecd841-a5d5-43db-bb17-4b05e897b59b","Type":"ContainerDied","Data":"db916b486b7338ed1893515212c3294dc2e37f9cce140bc336049ca7da5baebb"}
Oct 30 13:15:03.042701 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.042585    4993 scope.go:122] "RemoveContainer" containerID="db916b486b7338ed1893515212c3294dc2e37f9cce140bc336049ca7da5baebb"
Oct 30 13:15:03.045356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.045332    4993 generic.go:356] "Generic (PLEG): container finished" podID="dc13c75b-d11f-4444-af69-5db6694deea3" containerID="86c03c72d751bf334eb3f740da9b93d918aad357de836087a793a07a7be39fcf" exitCode=60
Oct 30 13:15:03.045596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.045560    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" event={"ID":"dc13c75b-d11f-4444-af69-5db6694deea3","Type":"ContainerDied","Data":"86c03c72d751bf334eb3f740da9b93d918aad357de836087a793a07a7be39fcf"}
Oct 30 13:15:03.045899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.045880    4993 scope.go:122] "RemoveContainer" containerID="86c03c72d751bf334eb3f740da9b93d918aad357de836087a793a07a7be39fcf"
Oct 30 13:15:03.052094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.051582    4993 generic.go:356] "Generic (PLEG): container finished" podID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerID="db9609621937c6f0353f68483d1d468640c68fd9be1328a03015b54ffb65ebff" exitCode=60
Oct 30 13:15:03.052094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.051613    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" event={"ID":"50a59e6b-4b6c-4bf4-b5d6-86462b68e611","Type":"ContainerDied","Data":"db9609621937c6f0353f68483d1d468640c68fd9be1328a03015b54ffb65ebff"}
Oct 30 13:15:03.052094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.051912    4993 scope.go:122] "RemoveContainer" containerID="db9609621937c6f0353f68483d1d468640c68fd9be1328a03015b54ffb65ebff"
Oct 30 13:15:03.699357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:03.699322    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a788667-becf-429a-af19-7df65728c6bd" path="/var/lib/kubelet/pods/1a788667-becf-429a-af19-7df65728c6bd/volumes"
Oct 30 13:15:04.072390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.071619    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" event={"ID":"50a59e6b-4b6c-4bf4-b5d6-86462b68e611","Type":"ContainerStarted","Data":"8648aa8bdb08f246753e4443f3cf6e4584784e37f4581c0983a865755d54b4ce"}
Oct 30 13:15:04.073974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.073935    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" event={"ID":"bb18ce19-72f1-42ad-8301-b8c5903f9aaa","Type":"ContainerStarted","Data":"5e1043512045f8c150fc9e29cc4665cfc5090b64845a3696235094a8aba4eceb"}
Oct 30 13:15:04.076595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.076566    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" event={"ID":"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759","Type":"ContainerStarted","Data":"7c37c6c2b8cbb0f7bb1974125d1b4de929ec056b10ba983626df11f242879fdf"}
Oct 30 13:15:04.078904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.078876    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" event={"ID":"f1116de9-a0c9-4e5e-974f-ac6f8734f32d","Type":"ContainerStarted","Data":"3d2a142a8daa58af3ce20c04018f4a4c2eb3ed5dc7d84f08af9e64903c5b01e0"}
Oct 30 13:15:04.081257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.081202    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" event={"ID":"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a","Type":"ContainerStarted","Data":"846bcf9dcf57846ce5f54e1eeae3a2e7833204bae9569b825c57bead6d5ecd0a"}
Oct 30 13:15:04.083665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.083634    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" event={"ID":"2976e31a-dc21-49b8-b7b1-31084a53d7e3","Type":"ContainerStarted","Data":"a7bc3cba3efac3011386051d26f793314910f710de56107bba95e76d3b1bda66"}
Oct 30 13:15:04.085895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.085874    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" event={"ID":"74dcbb92-946a-4c0f-9627-9acc70f32934","Type":"ContainerStarted","Data":"12869bcc05072358ac5908d16baf00763bf3ea4b8ca293b825ed486694afa02c"}
Oct 30 13:15:04.088066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.088042    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" event={"ID":"d3893102-450d-4e3d-974f-f57d67b662de","Type":"ContainerStarted","Data":"fcf15970ec360f2bcf3b464923fc1632558bb965512886bfd9359e8cdf7d0014"}
Oct 30 13:15:04.090479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.090334    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" event={"ID":"5ce95e7b-befb-4079-a2ab-b83cfd7f741d","Type":"ContainerStarted","Data":"06009f5215cd34da429bc39937ef2955a65fcf26b65b729923a0185a3686ff05"}
Oct 30 13:15:04.092749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.092728    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" event={"ID":"02ecd841-a5d5-43db-bb17-4b05e897b59b","Type":"ContainerStarted","Data":"f7c901ace0049458dad06d4541b5f92c1f33d8e8aae3e84543e75e8f8b054d87"}
Oct 30 13:15:04.094962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.094941    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" event={"ID":"dc13c75b-d11f-4444-af69-5db6694deea3","Type":"ContainerStarted","Data":"5005479ee56b3ec18cf846061b3af4c0f418925cd07f0455527c45bcfade2ea4"}
Oct 30 13:15:04.121856 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.121801    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" podStartSLOduration=4.121785658 podStartE2EDuration="4.121785658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.11807476 +0000 UTC m=+2568.593794370" watchObservedRunningTime="2026-10-30 13:15:04.121785658 +0000 UTC m=+2568.597505286"
Oct 30 13:15:04.166550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.166474    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" podStartSLOduration=4.166455481 podStartE2EDuration="4.166455481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.1625202 +0000 UTC m=+2568.638239810" watchObservedRunningTime="2026-10-30 13:15:04.166455481 +0000 UTC m=+2568.642175090"
Oct 30 13:15:04.309567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.309503    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" podStartSLOduration=4.309488519 podStartE2EDuration="4.309488519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.30450236 +0000 UTC m=+2568.780221966" watchObservedRunningTime="2026-10-30 13:15:04.309488519 +0000 UTC m=+2568.785208129"
Oct 30 13:15:04.348906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.348853    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" podStartSLOduration=4.348839033 podStartE2EDuration="4.348839033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.348405081 +0000 UTC m=+2568.824124691" watchObservedRunningTime="2026-10-30 13:15:04.348839033 +0000 UTC m=+2568.824558643"
Oct 30 13:15:04.443970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.443913    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" podStartSLOduration=4.443898722 podStartE2EDuration="4.443898722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.442346828 +0000 UTC m=+2568.918066419" watchObservedRunningTime="2026-10-30 13:15:04.443898722 +0000 UTC m=+2568.919618364"
Oct 30 13:15:04.494280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.494199    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" podStartSLOduration=4.49417985 podStartE2EDuration="4.49417985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.489662275 +0000 UTC m=+2568.965381915" watchObservedRunningTime="2026-10-30 13:15:04.49417985 +0000 UTC m=+2568.969899460"
Oct 30 13:15:04.535243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:04.535140    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" podStartSLOduration=4.535125435 podStartE2EDuration="4.535125435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.533100859 +0000 UTC m=+2569.008820470" watchObservedRunningTime="2026-10-30 13:15:04.535125435 +0000 UTC m=+2569.010845044"
Oct 30 13:15:05.108386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.108350    4993 generic.go:356] "Generic (PLEG): container finished" podID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerID="06009f5215cd34da429bc39937ef2955a65fcf26b65b729923a0185a3686ff05" exitCode=60
Oct 30 13:15:05.108888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.108436    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" event={"ID":"5ce95e7b-befb-4079-a2ab-b83cfd7f741d","Type":"ContainerDied","Data":"06009f5215cd34da429bc39937ef2955a65fcf26b65b729923a0185a3686ff05"}
Oct 30 13:15:05.108888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.108501    4993 scope.go:122] "RemoveContainer" containerID="6ee8a546b7af2e23366c7fb69e1ff844f3e72a648e4bce40de46d511b318e544"
Oct 30 13:15:05.108888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.108835    4993 scope.go:122] "RemoveContainer" containerID="06009f5215cd34da429bc39937ef2955a65fcf26b65b729923a0185a3686ff05"
Oct 30 13:15:05.109279 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.109257    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w_odh-ai-gateway-infra(5ce95e7b-befb-4079-a2ab-b83cfd7f741d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d"
Oct 30 13:15:05.110644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.110618    4993 generic.go:356] "Generic (PLEG): container finished" podID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerID="f7c901ace0049458dad06d4541b5f92c1f33d8e8aae3e84543e75e8f8b054d87" exitCode=60
Oct 30 13:15:05.110753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.110698    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" event={"ID":"02ecd841-a5d5-43db-bb17-4b05e897b59b","Type":"ContainerDied","Data":"f7c901ace0049458dad06d4541b5f92c1f33d8e8aae3e84543e75e8f8b054d87"}
Oct 30 13:15:05.110996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.110980    4993 scope.go:122] "RemoveContainer" containerID="f7c901ace0049458dad06d4541b5f92c1f33d8e8aae3e84543e75e8f8b054d87"
Oct 30 13:15:05.111414 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.111382    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g_odh-ai-gateway-infra(02ecd841-a5d5-43db-bb17-4b05e897b59b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b"
Oct 30 13:15:05.112890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.112864    4993 generic.go:356] "Generic (PLEG): container finished" podID="dc13c75b-d11f-4444-af69-5db6694deea3" containerID="5005479ee56b3ec18cf846061b3af4c0f418925cd07f0455527c45bcfade2ea4" exitCode=60
Oct 30 13:15:05.113007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.112909    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" event={"ID":"dc13c75b-d11f-4444-af69-5db6694deea3","Type":"ContainerDied","Data":"5005479ee56b3ec18cf846061b3af4c0f418925cd07f0455527c45bcfade2ea4"}
Oct 30 13:15:05.113413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.113366    4993 scope.go:122] "RemoveContainer" containerID="5005479ee56b3ec18cf846061b3af4c0f418925cd07f0455527c45bcfade2ea4"
Oct 30 13:15:05.113784 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.113757    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d_odh-ai-gateway-infra(dc13c75b-d11f-4444-af69-5db6694deea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" podUID="dc13c75b-d11f-4444-af69-5db6694deea3"
Oct 30 13:15:05.115288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.115264    4993 generic.go:356] "Generic (PLEG): container finished" podID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerID="8648aa8bdb08f246753e4443f3cf6e4584784e37f4581c0983a865755d54b4ce" exitCode=60
Oct 30 13:15:05.115393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.115353    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" event={"ID":"50a59e6b-4b6c-4bf4-b5d6-86462b68e611","Type":"ContainerDied","Data":"8648aa8bdb08f246753e4443f3cf6e4584784e37f4581c0983a865755d54b4ce"}
Oct 30 13:15:05.115622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.115602    4993 scope.go:122] "RemoveContainer" containerID="8648aa8bdb08f246753e4443f3cf6e4584784e37f4581c0983a865755d54b4ce"
Oct 30 13:15:05.115963 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.115938    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx_odh-ai-gateway-infra(50a59e6b-4b6c-4bf4-b5d6-86462b68e611)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611"
Oct 30 13:15:05.117692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.117671    4993 generic.go:356] "Generic (PLEG): container finished" podID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerID="5e1043512045f8c150fc9e29cc4665cfc5090b64845a3696235094a8aba4eceb" exitCode=60
Oct 30 13:15:05.117795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.117738    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" event={"ID":"bb18ce19-72f1-42ad-8301-b8c5903f9aaa","Type":"ContainerDied","Data":"5e1043512045f8c150fc9e29cc4665cfc5090b64845a3696235094a8aba4eceb"}
Oct 30 13:15:05.118017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.118002    4993 scope.go:122] "RemoveContainer" containerID="5e1043512045f8c150fc9e29cc4665cfc5090b64845a3696235094a8aba4eceb"
Oct 30 13:15:05.118349 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.118330    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt_odh-ai-gateway-infra(bb18ce19-72f1-42ad-8301-b8c5903f9aaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa"
Oct 30 13:15:05.119905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.119882    4993 generic.go:356] "Generic (PLEG): container finished" podID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerID="7c37c6c2b8cbb0f7bb1974125d1b4de929ec056b10ba983626df11f242879fdf" exitCode=60
Oct 30 13:15:05.120011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.119919    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" event={"ID":"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759","Type":"ContainerDied","Data":"7c37c6c2b8cbb0f7bb1974125d1b4de929ec056b10ba983626df11f242879fdf"}
Oct 30 13:15:05.120252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.120235    4993 scope.go:122] "RemoveContainer" containerID="7c37c6c2b8cbb0f7bb1974125d1b4de929ec056b10ba983626df11f242879fdf"
Oct 30 13:15:05.120557 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.120536    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh_odh-ai-gateway-infra(fa7ea7d5-a0ae-4f82-b657-8f5869d6f759)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759"
Oct 30 13:15:05.122157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.122134    4993 generic.go:356] "Generic (PLEG): container finished" podID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerID="3d2a142a8daa58af3ce20c04018f4a4c2eb3ed5dc7d84f08af9e64903c5b01e0" exitCode=60
Oct 30 13:15:05.122284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.122169    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" event={"ID":"f1116de9-a0c9-4e5e-974f-ac6f8734f32d","Type":"ContainerDied","Data":"3d2a142a8daa58af3ce20c04018f4a4c2eb3ed5dc7d84f08af9e64903c5b01e0"}
Oct 30 13:15:05.122602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.122581    4993 scope.go:122] "RemoveContainer" containerID="3d2a142a8daa58af3ce20c04018f4a4c2eb3ed5dc7d84f08af9e64903c5b01e0"
Oct 30 13:15:05.122933 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.122909    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p_odh-ai-gateway-infra(f1116de9-a0c9-4e5e-974f-ac6f8734f32d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d"
Oct 30 13:15:05.124489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.124466    4993 generic.go:356] "Generic (PLEG): container finished" podID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerID="846bcf9dcf57846ce5f54e1eeae3a2e7833204bae9569b825c57bead6d5ecd0a" exitCode=60
Oct 30 13:15:05.124606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.124563    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" event={"ID":"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a","Type":"ContainerDied","Data":"846bcf9dcf57846ce5f54e1eeae3a2e7833204bae9569b825c57bead6d5ecd0a"}
Oct 30 13:15:05.124787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.124768    4993 scope.go:122] "RemoveContainer" containerID="846bcf9dcf57846ce5f54e1eeae3a2e7833204bae9569b825c57bead6d5ecd0a"
Oct 30 13:15:05.124959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.124941    4993 scope.go:122] "RemoveContainer" containerID="db916b486b7338ed1893515212c3294dc2e37f9cce140bc336049ca7da5baebb"
Oct 30 13:15:05.125123 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.125058    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s_odh-ai-gateway-infra(c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a"
Oct 30 13:15:05.126765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.126746    4993 generic.go:356] "Generic (PLEG): container finished" podID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerID="a7bc3cba3efac3011386051d26f793314910f710de56107bba95e76d3b1bda66" exitCode=60
Oct 30 13:15:05.126885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.126827    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" event={"ID":"2976e31a-dc21-49b8-b7b1-31084a53d7e3","Type":"ContainerDied","Data":"a7bc3cba3efac3011386051d26f793314910f710de56107bba95e76d3b1bda66"}
Oct 30 13:15:05.127069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.127053    4993 scope.go:122] "RemoveContainer" containerID="a7bc3cba3efac3011386051d26f793314910f710de56107bba95e76d3b1bda66"
Oct 30 13:15:05.127387 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.127361    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq_odh-ai-gateway-infra(2976e31a-dc21-49b8-b7b1-31084a53d7e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3"
Oct 30 13:15:05.129141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.129119    4993 generic.go:356] "Generic (PLEG): container finished" podID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerID="12869bcc05072358ac5908d16baf00763bf3ea4b8ca293b825ed486694afa02c" exitCode=60
Oct 30 13:15:05.129395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.129362    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" event={"ID":"74dcbb92-946a-4c0f-9627-9acc70f32934","Type":"ContainerDied","Data":"12869bcc05072358ac5908d16baf00763bf3ea4b8ca293b825ed486694afa02c"}
Oct 30 13:15:05.129503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.129485    4993 scope.go:122] "RemoveContainer" containerID="12869bcc05072358ac5908d16baf00763bf3ea4b8ca293b825ed486694afa02c"
Oct 30 13:15:05.129781 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.129758    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm_odh-ai-gateway-infra(74dcbb92-946a-4c0f-9627-9acc70f32934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934"
Oct 30 13:15:05.131946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.131926    4993 generic.go:356] "Generic (PLEG): container finished" podID="d3893102-450d-4e3d-974f-f57d67b662de" containerID="fcf15970ec360f2bcf3b464923fc1632558bb965512886bfd9359e8cdf7d0014" exitCode=60
Oct 30 13:15:05.132134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.131999    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" event={"ID":"d3893102-450d-4e3d-974f-f57d67b662de","Type":"ContainerDied","Data":"fcf15970ec360f2bcf3b464923fc1632558bb965512886bfd9359e8cdf7d0014"}
Oct 30 13:15:05.132336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.132266    4993 scope.go:122] "RemoveContainer" containerID="fcf15970ec360f2bcf3b464923fc1632558bb965512886bfd9359e8cdf7d0014"
Oct 30 13:15:05.132574 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:05.132552    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6_odh-ai-gateway-infra(d3893102-450d-4e3d-974f-f57d67b662de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" podUID="d3893102-450d-4e3d-974f-f57d67b662de"
Oct 30 13:15:05.142330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.142299    4993 scope.go:122] "RemoveContainer" containerID="86c03c72d751bf334eb3f740da9b93d918aad357de836087a793a07a7be39fcf"
Oct 30 13:15:05.166426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.166390    4993 scope.go:122] "RemoveContainer" containerID="db9609621937c6f0353f68483d1d468640c68fd9be1328a03015b54ffb65ebff"
Oct 30 13:15:05.180474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.180445    4993 scope.go:122] "RemoveContainer" containerID="ebfd7500c8423e624fd26b8173eb9c2af59b9701209d4551cfb11bfe318cdb75"
Oct 30 13:15:05.202334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.202303    4993 scope.go:122] "RemoveContainer" containerID="0a0b0abd6971bb8d25b3586ad98737adb2da3e863586059507820196c609a70e"
Oct 30 13:15:05.217201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.217178    4993 scope.go:122] "RemoveContainer" containerID="a732492a91f4b4eae3432eaddc327f6600fa8ef9a4d03cf4dbe18b82f379f45d"
Oct 30 13:15:05.234158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.234134    4993 scope.go:122] "RemoveContainer" containerID="22354fae9915c562d918048a3f6c22cf08cd7ac1a8608cb11054814473d0f888"
Oct 30 13:15:05.249718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.249690    4993 scope.go:122] "RemoveContainer" containerID="90a6be2ad0022f528295f3f89592fce8a203f3753289f82e0baf7ad83031be82"
Oct 30 13:15:05.267983 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.267951    4993 scope.go:122] "RemoveContainer" containerID="b64f62543708649f6d9c38eae1d7e9720d74805c4dee424856c6ee6dab8ff122"
Oct 30 13:15:05.283653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:05.283623    4993 scope.go:122] "RemoveContainer" containerID="11ea79146c830cb977f41550b0f6eccbb04b7829db01e5bb841a5eaca2bcd64b"
Oct 30 13:15:06.147679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.147604    4993 scope.go:122] "RemoveContainer" containerID="8648aa8bdb08f246753e4443f3cf6e4584784e37f4581c0983a865755d54b4ce"
Oct 30 13:15:06.148080 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.147987    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx_odh-ai-gateway-infra(50a59e6b-4b6c-4bf4-b5d6-86462b68e611)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611"
Oct 30 13:15:06.150814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.150792    4993 scope.go:122] "RemoveContainer" containerID="5e1043512045f8c150fc9e29cc4665cfc5090b64845a3696235094a8aba4eceb"
Oct 30 13:15:06.151761 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.151733    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt_odh-ai-gateway-infra(bb18ce19-72f1-42ad-8301-b8c5903f9aaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa"
Oct 30 13:15:06.154153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.154129    4993 scope.go:122] "RemoveContainer" containerID="7c37c6c2b8cbb0f7bb1974125d1b4de929ec056b10ba983626df11f242879fdf"
Oct 30 13:15:06.154485 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.154464    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh_odh-ai-gateway-infra(fa7ea7d5-a0ae-4f82-b657-8f5869d6f759)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759"
Oct 30 13:15:06.156597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.156419    4993 scope.go:122] "RemoveContainer" containerID="3d2a142a8daa58af3ce20c04018f4a4c2eb3ed5dc7d84f08af9e64903c5b01e0"
Oct 30 13:15:06.156702 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.156669    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p_odh-ai-gateway-infra(f1116de9-a0c9-4e5e-974f-ac6f8734f32d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d"
Oct 30 13:15:06.161615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.161590    4993 scope.go:122] "RemoveContainer" containerID="846bcf9dcf57846ce5f54e1eeae3a2e7833204bae9569b825c57bead6d5ecd0a"
Oct 30 13:15:06.162890 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.162863    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s_odh-ai-gateway-infra(c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a"
Oct 30 13:15:06.164576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.164557    4993 scope.go:122] "RemoveContainer" containerID="a7bc3cba3efac3011386051d26f793314910f710de56107bba95e76d3b1bda66"
Oct 30 13:15:06.164838 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.164816    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq_odh-ai-gateway-infra(2976e31a-dc21-49b8-b7b1-31084a53d7e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3"
Oct 30 13:15:06.167024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.167001    4993 scope.go:122] "RemoveContainer" containerID="12869bcc05072358ac5908d16baf00763bf3ea4b8ca293b825ed486694afa02c"
Oct 30 13:15:06.167324 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.167299    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm_odh-ai-gateway-infra(74dcbb92-946a-4c0f-9627-9acc70f32934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934"
Oct 30 13:15:06.169296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.169277    4993 scope.go:122] "RemoveContainer" containerID="fcf15970ec360f2bcf3b464923fc1632558bb965512886bfd9359e8cdf7d0014"
Oct 30 13:15:06.169632 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.169591    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6_odh-ai-gateway-infra(d3893102-450d-4e3d-974f-f57d67b662de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" podUID="d3893102-450d-4e3d-974f-f57d67b662de"
Oct 30 13:15:06.171607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.171587    4993 scope.go:122] "RemoveContainer" containerID="06009f5215cd34da429bc39937ef2955a65fcf26b65b729923a0185a3686ff05"
Oct 30 13:15:06.171945 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.171924    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w_odh-ai-gateway-infra(5ce95e7b-befb-4079-a2ab-b83cfd7f741d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d"
Oct 30 13:15:06.174794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.174773    4993 scope.go:122] "RemoveContainer" containerID="f7c901ace0049458dad06d4541b5f92c1f33d8e8aae3e84543e75e8f8b054d87"
Oct 30 13:15:06.175163 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.175119    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g_odh-ai-gateway-infra(02ecd841-a5d5-43db-bb17-4b05e897b59b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b"
Oct 30 13:15:06.176653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:06.176630    4993 scope.go:122] "RemoveContainer" containerID="5005479ee56b3ec18cf846061b3af4c0f418925cd07f0455527c45bcfade2ea4"
Oct 30 13:15:06.176883 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:06.176857    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d_odh-ai-gateway-infra(dc13c75b-d11f-4444-af69-5db6694deea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" podUID="dc13c75b-d11f-4444-af69-5db6694deea3"
Oct 30 13:15:08.038556 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:08.038516    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"]
Oct 30 13:15:08.053561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:08.053526    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c6a7e3-px5v2"]
Oct 30 13:15:08.579267 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:08.579221    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:15:09.700229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:09.699425    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812684fe-ef9c-42d9-b140-960ad92a954a" path="/var/lib/kubelet/pods/812684fe-ef9c-42d9-b140-960ad92a954a/volumes"
Oct 30 13:15:11.042602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.042553    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"]
Oct 30 13:15:11.053883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.053847    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fad1f1aa-qm78l"]
Oct 30 13:15:11.124636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.124587    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"]
Oct 30 13:15:11.125413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.125389    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy"
Oct 30 13:15:11.125526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.125415    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy"
Oct 30 13:15:11.125581 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.125573    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a788667-becf-429a-af19-7df65728c6bd" containerName="istio-proxy"
Oct 30 13:15:11.129283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.129257    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.153463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.153418    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-71fbeb\"/\"openshift-service-ca.crt\""
Oct 30 13:15:11.153463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.153430    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-71fbeb\"/\"auth-test-model-71fbeb-kserve-self-signed-certs\""
Oct 30 13:15:11.153745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.153558    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-71fbeb\"/\"default-dockercfg-qdgkq\""
Oct 30 13:15:11.153894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.153864    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"]
Oct 30 13:15:11.154040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.153926    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-71fbeb\"/\"kube-root-ca.crt\""
Oct 30 13:15:11.258471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.258428    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-dshm\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.258656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.258505    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tmp-dir\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.258656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.258566    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-model-cache\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.258656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.258621    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-home\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.258879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.258807    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tls-certs\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.258879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.258832    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkvnn\" (UniqueName: \"kubernetes.io/projected/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-kube-api-access-bkvnn\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.360071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.359960    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-model-cache\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.360071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.360041    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-home\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.360352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.360284    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tls-certs\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.360352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.360310    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkvnn\" (UniqueName: \"kubernetes.io/projected/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-kube-api-access-bkvnn\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.360479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.360398    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-dshm\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.360479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.360451    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tmp-dir\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.360479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.360456    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-model-cache\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.360935 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.360911    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tmp-dir\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.361039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.360956    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-home\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.363416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.363380    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-dshm\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.364112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.364090    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tls-certs\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.389434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.389389    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkvnn\" (UniqueName: \"kubernetes.io/projected/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-kube-api-access-bkvnn\") pod \"auth-test-model-71fbeb-kserve-5b77f8997-59pzc\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") " pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.450420 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.449542    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:11.704542 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.702305    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0565be25-44d5-4648-b9af-81985b0cd8bc" path="/var/lib/kubelet/pods/0565be25-44d5-4648-b9af-81985b0cd8bc/volumes"
Oct 30 13:15:11.917435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:11.916366    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"]
Oct 30 13:15:11.921713 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:15:11.921680    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9d11f0f_84d6_462c_89f5_8c6ec6b06871.slice/crio-7ace8caaf5a370ab85298db2c1eb8be4f51f94170a3b0cad79cfe916abd7184c WatchSource:0}: Error finding container 7ace8caaf5a370ab85298db2c1eb8be4f51f94170a3b0cad79cfe916abd7184c: Status 404 returned error can't find the container with id 7ace8caaf5a370ab85298db2c1eb8be4f51f94170a3b0cad79cfe916abd7184c
Oct 30 13:15:12.268844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:12.265053    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc" event={"ID":"d9d11f0f-84d6-462c-89f5-8c6ec6b06871","Type":"ContainerStarted","Data":"d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a"}
Oct 30 13:15:12.270752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:12.270708    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc" event={"ID":"d9d11f0f-84d6-462c-89f5-8c6ec6b06871","Type":"ContainerStarted","Data":"7ace8caaf5a370ab85298db2c1eb8be4f51f94170a3b0cad79cfe916abd7184c"}
Oct 30 13:15:12.321431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:12.316546    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc" podStartSLOduration=1.316525618 podStartE2EDuration="1.316525618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:12.315120036 +0000 UTC m=+2576.790839646" watchObservedRunningTime="2026-10-30 13:15:12.316525618 +0000 UTC m=+2576.792245229"
Oct 30 13:15:12.451137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:12.451099    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:14.684890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:14.684855    4993 scope.go:122] "RemoveContainer" containerID="846bcf9dcf57846ce5f54e1eeae3a2e7833204bae9569b825c57bead6d5ecd0a"
Oct 30 13:15:14.685362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:14.685024    4993 scope.go:122] "RemoveContainer" containerID="fcf15970ec360f2bcf3b464923fc1632558bb965512886bfd9359e8cdf7d0014"
Oct 30 13:15:14.685362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:14.685081    4993 scope.go:122] "RemoveContainer" containerID="06009f5215cd34da429bc39937ef2955a65fcf26b65b729923a0185a3686ff05"
Oct 30 13:15:15.302183 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:15.302139    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" event={"ID":"d3893102-450d-4e3d-974f-f57d67b662de","Type":"ContainerStarted","Data":"b20b6d82cb34b820c251c8e09bb344705b9e6961824413bd530d51b672b245ff"}
Oct 30 13:15:15.304390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:15.304360    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" event={"ID":"5ce95e7b-befb-4079-a2ab-b83cfd7f741d","Type":"ContainerStarted","Data":"c9ece3e119ddfb77922b88fe8a58ed6c04d6c530df1237a4e3299be69a6be5f7"}
Oct 30 13:15:15.306520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:15.306497    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" event={"ID":"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a","Type":"ContainerStarted","Data":"d4e57e80eb09afafea3c26fae10a5f22d0af1e10daac685f313688f379606cf1"}
Oct 30 13:15:15.689892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:15.689802    4993 scope.go:122] "RemoveContainer" containerID="5005479ee56b3ec18cf846061b3af4c0f418925cd07f0455527c45bcfade2ea4"
Oct 30 13:15:16.317718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.317681    4993 generic.go:356] "Generic (PLEG): container finished" podID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerID="c9ece3e119ddfb77922b88fe8a58ed6c04d6c530df1237a4e3299be69a6be5f7" exitCode=60
Oct 30 13:15:16.317922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.317745    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" event={"ID":"5ce95e7b-befb-4079-a2ab-b83cfd7f741d","Type":"ContainerDied","Data":"c9ece3e119ddfb77922b88fe8a58ed6c04d6c530df1237a4e3299be69a6be5f7"}
Oct 30 13:15:16.317922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.317832    4993 scope.go:122] "RemoveContainer" containerID="06009f5215cd34da429bc39937ef2955a65fcf26b65b729923a0185a3686ff05"
Oct 30 13:15:16.318370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.318349    4993 scope.go:122] "RemoveContainer" containerID="c9ece3e119ddfb77922b88fe8a58ed6c04d6c530df1237a4e3299be69a6be5f7"
Oct 30 13:15:16.318755 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:16.318731    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w_odh-ai-gateway-infra(5ce95e7b-befb-4079-a2ab-b83cfd7f741d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d"
Oct 30 13:15:16.320036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.320017    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" event={"ID":"dc13c75b-d11f-4444-af69-5db6694deea3","Type":"ContainerStarted","Data":"3777fca81c3ae28d7f110c45615dcda141d85a1a97e0be2275510c378c26d58c"}
Oct 30 13:15:16.322160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.322132    4993 generic.go:356] "Generic (PLEG): container finished" podID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerID="d4e57e80eb09afafea3c26fae10a5f22d0af1e10daac685f313688f379606cf1" exitCode=60
Oct 30 13:15:16.322249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.322201    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" event={"ID":"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a","Type":"ContainerDied","Data":"d4e57e80eb09afafea3c26fae10a5f22d0af1e10daac685f313688f379606cf1"}
Oct 30 13:15:16.322614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.322594    4993 scope.go:122] "RemoveContainer" containerID="d4e57e80eb09afafea3c26fae10a5f22d0af1e10daac685f313688f379606cf1"
Oct 30 13:15:16.322937 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:16.322915    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s_odh-ai-gateway-infra(c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a"
Oct 30 13:15:16.324358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.324338    4993 generic.go:356] "Generic (PLEG): container finished" podID="d3893102-450d-4e3d-974f-f57d67b662de" containerID="b20b6d82cb34b820c251c8e09bb344705b9e6961824413bd530d51b672b245ff" exitCode=60
Oct 30 13:15:16.324457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.324378    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" event={"ID":"d3893102-450d-4e3d-974f-f57d67b662de","Type":"ContainerDied","Data":"b20b6d82cb34b820c251c8e09bb344705b9e6961824413bd530d51b672b245ff"}
Oct 30 13:15:16.324709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.324693    4993 scope.go:122] "RemoveContainer" containerID="b20b6d82cb34b820c251c8e09bb344705b9e6961824413bd530d51b672b245ff"
Oct 30 13:15:16.324973 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:16.324954    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6_odh-ai-gateway-infra(d3893102-450d-4e3d-974f-f57d67b662de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" podUID="d3893102-450d-4e3d-974f-f57d67b662de"
Oct 30 13:15:16.332447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.332424    4993 scope.go:122] "RemoveContainer" containerID="846bcf9dcf57846ce5f54e1eeae3a2e7833204bae9569b825c57bead6d5ecd0a"
Oct 30 13:15:16.346843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.346820    4993 scope.go:122] "RemoveContainer" containerID="fcf15970ec360f2bcf3b464923fc1632558bb965512886bfd9359e8cdf7d0014"
Oct 30 13:15:16.365557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.365527    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"]
Oct 30 13:15:16.407011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.406971    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"]
Oct 30 13:15:16.453215 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.453162    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"]
Oct 30 13:15:16.684920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.684837    4993 scope.go:122] "RemoveContainer" containerID="7c37c6c2b8cbb0f7bb1974125d1b4de929ec056b10ba983626df11f242879fdf"
Oct 30 13:15:16.685088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.685027    4993 scope.go:122] "RemoveContainer" containerID="a7bc3cba3efac3011386051d26f793314910f710de56107bba95e76d3b1bda66"
Oct 30 13:15:16.685161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:16.685087    4993 scope.go:122] "RemoveContainer" containerID="8648aa8bdb08f246753e4443f3cf6e4584784e37f4581c0983a865755d54b4ce"
Oct 30 13:15:17.336951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.336911    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" event={"ID":"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759","Type":"ContainerStarted","Data":"374390f84c8546b243a9c209bef568da6da140daf4659cc8f9a992d96cd82e6d"}
Oct 30 13:15:17.342541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.342480    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" event={"ID":"2976e31a-dc21-49b8-b7b1-31084a53d7e3","Type":"ContainerStarted","Data":"0fe74012dd5de0cb7edc0e9a23187be087ccb85ac35aced6479db54fc919cb46"}
Oct 30 13:15:17.349497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.349450    4993 generic.go:356] "Generic (PLEG): container finished" podID="dc13c75b-d11f-4444-af69-5db6694deea3" containerID="3777fca81c3ae28d7f110c45615dcda141d85a1a97e0be2275510c378c26d58c" exitCode=60
Oct 30 13:15:17.349610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.349511    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" event={"ID":"dc13c75b-d11f-4444-af69-5db6694deea3","Type":"ContainerDied","Data":"3777fca81c3ae28d7f110c45615dcda141d85a1a97e0be2275510c378c26d58c"}
Oct 30 13:15:17.349610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.349580    4993 scope.go:122] "RemoveContainer" containerID="5005479ee56b3ec18cf846061b3af4c0f418925cd07f0455527c45bcfade2ea4"
Oct 30 13:15:17.350061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.350044    4993 scope.go:122] "RemoveContainer" containerID="3777fca81c3ae28d7f110c45615dcda141d85a1a97e0be2275510c378c26d58c"
Oct 30 13:15:17.350482 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:17.350457    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d_odh-ai-gateway-infra(dc13c75b-d11f-4444-af69-5db6694deea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" podUID="dc13c75b-d11f-4444-af69-5db6694deea3"
Oct 30 13:15:17.352465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.352289    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" event={"ID":"50a59e6b-4b6c-4bf4-b5d6-86462b68e611","Type":"ContainerStarted","Data":"30084f77272437dc45adaccc8fc5b958567b346221da3e25dae03ff1b5119d93"}
Oct 30 13:15:17.497093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.497052    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"]
Oct 30 13:15:17.634786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.634697    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:17.707381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.705127    4993 scope.go:122] "RemoveContainer" containerID="3d2a142a8daa58af3ce20c04018f4a4c2eb3ed5dc7d84f08af9e64903c5b01e0"
Oct 30 13:15:17.708628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.708599    4993 scope.go:122] "RemoveContainer" containerID="f7c901ace0049458dad06d4541b5f92c1f33d8e8aae3e84543e75e8f8b054d87"
Oct 30 13:15:17.729272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.729242    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:17.766133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.766094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-maas-api-service-ca\") pod \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\" (UID: \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\") "
Oct 30 13:15:17.766457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.766432    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-kube-api-access-k64tv\" (UniqueName: \"kubernetes.io/projected/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-kube-api-access-k64tv\") pod \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\" (UID: \"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a\") "
Oct 30 13:15:17.782415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.766560    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-maas-api-service-ca" pod "c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" (UID: "c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:17.782415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.771149    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:17.797508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.797466    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-kube-api-access-k64tv" pod "c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" (UID: "c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a"). InnerVolumeSpecName "kube-api-access-k64tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:17.817252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.816688    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:17.871795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.871760    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-maas-api-service-ca\") pod \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\" (UID: \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\") "
Oct 30 13:15:17.872123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.872101    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-kube-api-access-kzfph\" (UniqueName: \"kubernetes.io/projected/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-kube-api-access-kzfph\") pod \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\" (UID: \"5ce95e7b-befb-4079-a2ab-b83cfd7f741d\") "
Oct 30 13:15:17.872476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.872148    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-maas-api-service-ca" pod "5ce95e7b-befb-4079-a2ab-b83cfd7f741d" (UID: "5ce95e7b-befb-4079-a2ab-b83cfd7f741d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:17.874593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.874563    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k64tv\" (UniqueName: \"kubernetes.io/projected/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a-kube-api-access-k64tv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:17.874593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.874586    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:17.876553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.876526    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-kube-api-access-kzfph" pod "5ce95e7b-befb-4079-a2ab-b83cfd7f741d" (UID: "5ce95e7b-befb-4079-a2ab-b83cfd7f741d"). InnerVolumeSpecName "kube-api-access-kzfph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:17.975461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.975415    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3893102-450d-4e3d-974f-f57d67b662de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3893102-450d-4e3d-974f-f57d67b662de-maas-api-service-ca\") pod \"d3893102-450d-4e3d-974f-f57d67b662de\" (UID: \"d3893102-450d-4e3d-974f-f57d67b662de\") "
Oct 30 13:15:17.975664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.975576    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3893102-450d-4e3d-974f-f57d67b662de-kube-api-access-xlrxt\" (UniqueName: \"kubernetes.io/projected/d3893102-450d-4e3d-974f-f57d67b662de-kube-api-access-xlrxt\") pod \"d3893102-450d-4e3d-974f-f57d67b662de\" (UID: \"d3893102-450d-4e3d-974f-f57d67b662de\") "
Oct 30 13:15:17.975879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.975848    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3893102-450d-4e3d-974f-f57d67b662de-maas-api-service-ca" pod "d3893102-450d-4e3d-974f-f57d67b662de" (UID: "d3893102-450d-4e3d-974f-f57d67b662de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:17.976290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.976268    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzfph\" (UniqueName: \"kubernetes.io/projected/5ce95e7b-befb-4079-a2ab-b83cfd7f741d-kube-api-access-kzfph\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:17.976366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.976294    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3893102-450d-4e3d-974f-f57d67b662de-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:17.977754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:17.977725    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3893102-450d-4e3d-974f-f57d67b662de-kube-api-access-xlrxt" pod "d3893102-450d-4e3d-974f-f57d67b662de" (UID: "d3893102-450d-4e3d-974f-f57d67b662de"). InnerVolumeSpecName "kube-api-access-xlrxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:18.077809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.077769    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrxt\" (UniqueName: \"kubernetes.io/projected/d3893102-450d-4e3d-974f-f57d67b662de-kube-api-access-xlrxt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:18.366915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.366877    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" event={"ID":"f1116de9-a0c9-4e5e-974f-ac6f8734f32d","Type":"ContainerStarted","Data":"a7cd004239fc97d399121c20b858e14859a6be77d7fc84d33445713fbb133825"}
Oct 30 13:15:18.368911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.368871    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s" event={"ID":"c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a","Type":"ContainerDied","Data":"85f85358721fdb879d6bf37e27e97af6e00d6c5357ad4598aab457a2755f7c13"}
Oct 30 13:15:18.368911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.368897    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"
Oct 30 13:15:18.369123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.368953    4993 scope.go:122] "RemoveContainer" containerID="d4e57e80eb09afafea3c26fae10a5f22d0af1e10daac685f313688f379606cf1"
Oct 30 13:15:18.371526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.371493    4993 generic.go:356] "Generic (PLEG): container finished" podID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerID="0fe74012dd5de0cb7edc0e9a23187be087ccb85ac35aced6479db54fc919cb46" exitCode=60
Oct 30 13:15:18.371662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.371536    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" event={"ID":"2976e31a-dc21-49b8-b7b1-31084a53d7e3","Type":"ContainerDied","Data":"0fe74012dd5de0cb7edc0e9a23187be087ccb85ac35aced6479db54fc919cb46"}
Oct 30 13:15:18.372038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.372021    4993 scope.go:122] "RemoveContainer" containerID="0fe74012dd5de0cb7edc0e9a23187be087ccb85ac35aced6479db54fc919cb46"
Oct 30 13:15:18.372458 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:18.372427    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq_odh-ai-gateway-infra(2976e31a-dc21-49b8-b7b1-31084a53d7e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3"
Oct 30 13:15:18.373592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.373572    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"
Oct 30 13:15:18.373696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.373564    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6" event={"ID":"d3893102-450d-4e3d-974f-f57d67b662de","Type":"ContainerDied","Data":"86bddfb144fbc642c9dd61fae972ea7fd83855f63c67fcc7e2eb965da7004aba"}
Oct 30 13:15:18.376247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.375649    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"
Oct 30 13:15:18.376247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.375652    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w" event={"ID":"5ce95e7b-befb-4079-a2ab-b83cfd7f741d","Type":"ContainerDied","Data":"9d9b0a906f28a08b757eca9d3be0ed332ab141807b3d0100d24fde7cd81f07f6"}
Oct 30 13:15:18.380003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.379930    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" event={"ID":"02ecd841-a5d5-43db-bb17-4b05e897b59b","Type":"ContainerStarted","Data":"08d9c97cbfba4a61a275e3e49a28843f68f168f30423d02f2286d34eba2b944c"}
Oct 30 13:15:18.387078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.387055    4993 scope.go:122] "RemoveContainer" containerID="a7bc3cba3efac3011386051d26f793314910f710de56107bba95e76d3b1bda66"
Oct 30 13:15:18.392608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.392581    4993 generic.go:356] "Generic (PLEG): container finished" podID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerID="30084f77272437dc45adaccc8fc5b958567b346221da3e25dae03ff1b5119d93" exitCode=60
Oct 30 13:15:18.392732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.392655    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" event={"ID":"50a59e6b-4b6c-4bf4-b5d6-86462b68e611","Type":"ContainerDied","Data":"30084f77272437dc45adaccc8fc5b958567b346221da3e25dae03ff1b5119d93"}
Oct 30 13:15:18.393134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.393114    4993 scope.go:122] "RemoveContainer" containerID="30084f77272437dc45adaccc8fc5b958567b346221da3e25dae03ff1b5119d93"
Oct 30 13:15:18.393568 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:18.393544    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx_odh-ai-gateway-infra(50a59e6b-4b6c-4bf4-b5d6-86462b68e611)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611"
Oct 30 13:15:18.395397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.395372    4993 generic.go:356] "Generic (PLEG): container finished" podID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerID="374390f84c8546b243a9c209bef568da6da140daf4659cc8f9a992d96cd82e6d" exitCode=60
Oct 30 13:15:18.395499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.395429    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" event={"ID":"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759","Type":"ContainerDied","Data":"374390f84c8546b243a9c209bef568da6da140daf4659cc8f9a992d96cd82e6d"}
Oct 30 13:15:18.395845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.395827    4993 scope.go:122] "RemoveContainer" containerID="374390f84c8546b243a9c209bef568da6da140daf4659cc8f9a992d96cd82e6d"
Oct 30 13:15:18.396243 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:18.396187    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh_odh-ai-gateway-infra(fa7ea7d5-a0ae-4f82-b657-8f5869d6f759)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759"
Oct 30 13:15:18.405393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.405364    4993 scope.go:122] "RemoveContainer" containerID="b20b6d82cb34b820c251c8e09bb344705b9e6961824413bd530d51b672b245ff"
Oct 30 13:15:18.406653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.406627    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"]
Oct 30 13:15:18.422994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.422970    4993 scope.go:122] "RemoveContainer" containerID="c9ece3e119ddfb77922b88fe8a58ed6c04d6c530df1237a4e3299be69a6be5f7"
Oct 30 13:15:18.438142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.438112    4993 scope.go:122] "RemoveContainer" containerID="8648aa8bdb08f246753e4443f3cf6e4584784e37f4581c0983a865755d54b4ce"
Oct 30 13:15:18.460744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.460710    4993 scope.go:122] "RemoveContainer" containerID="7c37c6c2b8cbb0f7bb1974125d1b4de929ec056b10ba983626df11f242879fdf"
Oct 30 13:15:18.523328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.523293    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"]
Oct 30 13:15:18.574801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.574770    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:18.575991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.575962    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"]
Oct 30 13:15:18.584706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.584671    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889435-pztc6"]
Oct 30 13:15:18.661838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.661792    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"]
Oct 30 13:15:18.667922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.667890    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889435-j7g8w"]
Oct 30 13:15:18.684167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.684009    4993 scope.go:122] "RemoveContainer" containerID="12869bcc05072358ac5908d16baf00763bf3ea4b8ca293b825ed486694afa02c"
Oct 30 13:15:18.684407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.684385    4993 scope.go:122] "RemoveContainer" containerID="5e1043512045f8c150fc9e29cc4665cfc5090b64845a3696235094a8aba4eceb"
Oct 30 13:15:18.686581 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.686559    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc13c75b-d11f-4444-af69-5db6694deea3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc13c75b-d11f-4444-af69-5db6694deea3-maas-api-service-ca\") pod \"dc13c75b-d11f-4444-af69-5db6694deea3\" (UID: \"dc13c75b-d11f-4444-af69-5db6694deea3\") "
Oct 30 13:15:18.686703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.686689    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc13c75b-d11f-4444-af69-5db6694deea3-kube-api-access-5c9hl\" (UniqueName: \"kubernetes.io/projected/dc13c75b-d11f-4444-af69-5db6694deea3-kube-api-access-5c9hl\") pod \"dc13c75b-d11f-4444-af69-5db6694deea3\" (UID: \"dc13c75b-d11f-4444-af69-5db6694deea3\") "
Oct 30 13:15:18.686996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.686969    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc13c75b-d11f-4444-af69-5db6694deea3-maas-api-service-ca" pod "dc13c75b-d11f-4444-af69-5db6694deea3" (UID: "dc13c75b-d11f-4444-af69-5db6694deea3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:18.687109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.687077    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc13c75b-d11f-4444-af69-5db6694deea3-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:18.690391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.690364    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc13c75b-d11f-4444-af69-5db6694deea3-kube-api-access-5c9hl" pod "dc13c75b-d11f-4444-af69-5db6694deea3" (UID: "dc13c75b-d11f-4444-af69-5db6694deea3"). InnerVolumeSpecName "kube-api-access-5c9hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:18.698039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.698010    4993 scope.go:122] "RemoveContainer" containerID="5bea54239b20473c901b5c468096bb0910686ccd9400aca1d3dd242d48252346"
Oct 30 13:15:18.714625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.714592    4993 scope.go:122] "RemoveContainer" containerID="dbfdd85cb83656ab1acba5d4f561a575f667a61f0eebebd581a3d49d84d1bd2b"
Oct 30 13:15:18.717633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.717606    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"]
Oct 30 13:15:18.732800 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.732778    4993 scope.go:122] "RemoveContainer" containerID="9fcac392cc0d7ab9c9e05d4448d4ef4de078e366742d76fd1640cecbee2d2e9f"
Oct 30 13:15:18.738624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.738598    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-2fdf91-29889435-h4j6s"]
Oct 30 13:15:18.769274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.763430    4993 scope.go:122] "RemoveContainer" containerID="e6b0cfe4ec6b253d6e9eaf53fc2a849f0bc8d23101090cb58acbd929eb732bd9"
Oct 30 13:15:18.769274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.768695    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"]
Oct 30 13:15:18.788600 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.788562    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c9hl\" (UniqueName: \"kubernetes.io/projected/dc13c75b-d11f-4444-af69-5db6694deea3-kube-api-access-5c9hl\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:18.826867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:18.826837    4993 scope.go:122] "RemoveContainer" containerID="c39e510d3f5edd4cc31f37d08e8fa6f228a2f729168c5c9dc23d764f9f619a8a"
Oct 30 13:15:19.407146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.407109    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" event={"ID":"bb18ce19-72f1-42ad-8301-b8c5903f9aaa","Type":"ContainerStarted","Data":"b09aa8d55b547d26480e9a328a4eb818239a70d5f084c444bcedd0413deffad7"}
Oct 30 13:15:19.411925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.411897    4993 generic.go:356] "Generic (PLEG): container finished" podID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerID="a7cd004239fc97d399121c20b858e14859a6be77d7fc84d33445713fbb133825" exitCode=60
Oct 30 13:15:19.412057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.411929    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" event={"ID":"f1116de9-a0c9-4e5e-974f-ac6f8734f32d","Type":"ContainerDied","Data":"a7cd004239fc97d399121c20b858e14859a6be77d7fc84d33445713fbb133825"}
Oct 30 13:15:19.412057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.411991    4993 scope.go:122] "RemoveContainer" containerID="3d2a142a8daa58af3ce20c04018f4a4c2eb3ed5dc7d84f08af9e64903c5b01e0"
Oct 30 13:15:19.412455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.412435    4993 scope.go:122] "RemoveContainer" containerID="a7cd004239fc97d399121c20b858e14859a6be77d7fc84d33445713fbb133825"
Oct 30 13:15:19.412867 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:19.412843    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p_odh-ai-gateway-infra(f1116de9-a0c9-4e5e-974f-ac6f8734f32d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d"
Oct 30 13:15:19.416797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.416763    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" event={"ID":"74dcbb92-946a-4c0f-9627-9acc70f32934","Type":"ContainerStarted","Data":"a86d2faeaf08f44e4e9797c6fddc82049ae2eb1b374f67098043f884642523ab"}
Oct 30 13:15:19.431457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.431303    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"]
Oct 30 13:15:19.439645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.439604    4993 generic.go:356] "Generic (PLEG): container finished" podID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerID="08d9c97cbfba4a61a275e3e49a28843f68f168f30423d02f2286d34eba2b944c" exitCode=60
Oct 30 13:15:19.439821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.439773    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" event={"ID":"02ecd841-a5d5-43db-bb17-4b05e897b59b","Type":"ContainerDied","Data":"08d9c97cbfba4a61a275e3e49a28843f68f168f30423d02f2286d34eba2b944c"}
Oct 30 13:15:19.440333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.440298    4993 scope.go:122] "RemoveContainer" containerID="08d9c97cbfba4a61a275e3e49a28843f68f168f30423d02f2286d34eba2b944c"
Oct 30 13:15:19.440802 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:19.440779    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g_odh-ai-gateway-infra(02ecd841-a5d5-43db-bb17-4b05e897b59b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b"
Oct 30 13:15:19.445727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.445701    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"
Oct 30 13:15:19.445727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.445716    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d" event={"ID":"dc13c75b-d11f-4444-af69-5db6694deea3","Type":"ContainerDied","Data":"7788f6fa95963a846f00e5f7fee7869e0515c60bdedc53a86215028fde1890b3"}
Oct 30 13:15:19.469413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.469345    4993 scope.go:122] "RemoveContainer" containerID="f7c901ace0049458dad06d4541b5f92c1f33d8e8aae3e84543e75e8f8b054d87"
Oct 30 13:15:19.487516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.487482    4993 scope.go:122] "RemoveContainer" containerID="3777fca81c3ae28d7f110c45615dcda141d85a1a97e0be2275510c378c26d58c"
Oct 30 13:15:19.700679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.700645    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" path="/var/lib/kubelet/pods/5ce95e7b-befb-4079-a2ab-b83cfd7f741d/volumes"
Oct 30 13:15:19.701667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.701641    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" path="/var/lib/kubelet/pods/c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a/volumes"
Oct 30 13:15:19.702474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.702450    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3893102-450d-4e3d-974f-f57d67b662de" path="/var/lib/kubelet/pods/d3893102-450d-4e3d-974f-f57d67b662de/volumes"
Oct 30 13:15:19.760708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.760645    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:19.812917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.812880    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"]
Oct 30 13:15:19.864457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.864424    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:19.867261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.867146    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:19.868278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.868193    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"]
Oct 30 13:15:19.875992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.875957    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889435-k5p2d"]
Oct 30 13:15:19.910781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.910747    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-kube-api-access-v8rsn\" (UniqueName: \"kubernetes.io/projected/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-kube-api-access-v8rsn\") pod \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\" (UID: \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\") "
Oct 30 13:15:19.911044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.911019    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-maas-api-service-ca\") pod \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\" (UID: \"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759\") "
Oct 30 13:15:19.911668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.911628    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-maas-api-service-ca" pod "fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" (UID: "fa7ea7d5-a0ae-4f82-b657-8f5869d6f759"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:19.913962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:19.913937    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-kube-api-access-v8rsn" pod "fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" (UID: "fa7ea7d5-a0ae-4f82-b657-8f5869d6f759"). InnerVolumeSpecName "kube-api-access-v8rsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.013025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.012989    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2976e31a-dc21-49b8-b7b1-31084a53d7e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976e31a-dc21-49b8-b7b1-31084a53d7e3-maas-api-service-ca\") pod \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\" (UID: \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\") "
Oct 30 13:15:20.013188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.013056    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2976e31a-dc21-49b8-b7b1-31084a53d7e3-kube-api-access-596d4\" (UniqueName: \"kubernetes.io/projected/2976e31a-dc21-49b8-b7b1-31084a53d7e3-kube-api-access-596d4\") pod \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\" (UID: \"2976e31a-dc21-49b8-b7b1-31084a53d7e3\") "
Oct 30 13:15:20.013254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.013228    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-kube-api-access-9wjnt\" (UniqueName: \"kubernetes.io/projected/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-kube-api-access-9wjnt\") pod \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\" (UID: \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\") "
Oct 30 13:15:20.013351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.013333    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-maas-api-service-ca\") pod \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\" (UID: \"50a59e6b-4b6c-4bf4-b5d6-86462b68e611\") "
Oct 30 13:15:20.013423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.013378    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2976e31a-dc21-49b8-b7b1-31084a53d7e3-maas-api-service-ca" pod "2976e31a-dc21-49b8-b7b1-31084a53d7e3" (UID: "2976e31a-dc21-49b8-b7b1-31084a53d7e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.013721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.013691    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-maas-api-service-ca" pod "50a59e6b-4b6c-4bf4-b5d6-86462b68e611" (UID: "50a59e6b-4b6c-4bf4-b5d6-86462b68e611"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.014018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.013998    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.014165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.014024    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.014165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.014040    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8rsn\" (UniqueName: \"kubernetes.io/projected/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759-kube-api-access-v8rsn\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.014165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.014054    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976e31a-dc21-49b8-b7b1-31084a53d7e3-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.015571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.015545    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-kube-api-access-9wjnt" pod "50a59e6b-4b6c-4bf4-b5d6-86462b68e611" (UID: "50a59e6b-4b6c-4bf4-b5d6-86462b68e611"). InnerVolumeSpecName "kube-api-access-9wjnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.015678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.015635    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2976e31a-dc21-49b8-b7b1-31084a53d7e3-kube-api-access-596d4" pod "2976e31a-dc21-49b8-b7b1-31084a53d7e3" (UID: "2976e31a-dc21-49b8-b7b1-31084a53d7e3"). InnerVolumeSpecName "kube-api-access-596d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.116005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.115970    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wjnt\" (UniqueName: \"kubernetes.io/projected/50a59e6b-4b6c-4bf4-b5d6-86462b68e611-kube-api-access-9wjnt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.116005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.116002    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596d4\" (UniqueName: \"kubernetes.io/projected/2976e31a-dc21-49b8-b7b1-31084a53d7e3-kube-api-access-596d4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.463709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.463673    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"
Oct 30 13:15:20.463709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.463690    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx" event={"ID":"50a59e6b-4b6c-4bf4-b5d6-86462b68e611","Type":"ContainerDied","Data":"35444618a08f1dcad53a57f553b16b897057b6118c83f151b80d67813329f180"}
Oct 30 13:15:20.464190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.463751    4993 scope.go:122] "RemoveContainer" containerID="30084f77272437dc45adaccc8fc5b958567b346221da3e25dae03ff1b5119d93"
Oct 30 13:15:20.466499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.466469    4993 generic.go:356] "Generic (PLEG): container finished" podID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerID="b09aa8d55b547d26480e9a328a4eb818239a70d5f084c444bcedd0413deffad7" exitCode=60
Oct 30 13:15:20.466726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.466701    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" event={"ID":"bb18ce19-72f1-42ad-8301-b8c5903f9aaa","Type":"ContainerDied","Data":"b09aa8d55b547d26480e9a328a4eb818239a70d5f084c444bcedd0413deffad7"}
Oct 30 13:15:20.467158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.467133    4993 scope.go:122] "RemoveContainer" containerID="b09aa8d55b547d26480e9a328a4eb818239a70d5f084c444bcedd0413deffad7"
Oct 30 13:15:20.467496 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:20.467474    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt_odh-ai-gateway-infra(bb18ce19-72f1-42ad-8301-b8c5903f9aaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa"
Oct 30 13:15:20.469577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.469512    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"
Oct 30 13:15:20.469577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.469541    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh" event={"ID":"fa7ea7d5-a0ae-4f82-b657-8f5869d6f759","Type":"ContainerDied","Data":"93879694d9014dafac04d1fac8badc8cfecdf6ff64ea249392427b04da0f40c4"}
Oct 30 13:15:20.472968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.472937    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"]
Oct 30 13:15:20.478348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.474558    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq" event={"ID":"2976e31a-dc21-49b8-b7b1-31084a53d7e3","Type":"ContainerDied","Data":"5cf77089bbe87542e4a53cd8049dbd7ac87825f0bd78c86bbacedf3b5e637202"}
Oct 30 13:15:20.478348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.474664    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"
Oct 30 13:15:20.481959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.481924    4993 scope.go:122] "RemoveContainer" containerID="5e1043512045f8c150fc9e29cc4665cfc5090b64845a3696235094a8aba4eceb"
Oct 30 13:15:20.547607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.547570    4993 generic.go:356] "Generic (PLEG): container finished" podID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerID="a86d2faeaf08f44e4e9797c6fddc82049ae2eb1b374f67098043f884642523ab" exitCode=60
Oct 30 13:15:20.547784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.547669    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" event={"ID":"74dcbb92-946a-4c0f-9627-9acc70f32934","Type":"ContainerDied","Data":"a86d2faeaf08f44e4e9797c6fddc82049ae2eb1b374f67098043f884642523ab"}
Oct 30 13:15:20.548177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.548152    4993 scope.go:122] "RemoveContainer" containerID="a86d2faeaf08f44e4e9797c6fddc82049ae2eb1b374f67098043f884642523ab"
Oct 30 13:15:20.560701 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:20.560659    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm_odh-ai-gateway-infra(74dcbb92-946a-4c0f-9627-9acc70f32934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934"
Oct 30 13:15:20.567743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.567710    4993 scope.go:122] "RemoveContainer" containerID="374390f84c8546b243a9c209bef568da6da140daf4659cc8f9a992d96cd82e6d"
Oct 30 13:15:20.569280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.569252    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"]
Oct 30 13:15:20.597613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.597492    4993 scope.go:122] "RemoveContainer" containerID="0fe74012dd5de0cb7edc0e9a23187be087ccb85ac35aced6479db54fc919cb46"
Oct 30 13:15:20.637560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.637529    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"]
Oct 30 13:15:20.643772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.643747    4993 scope.go:122] "RemoveContainer" containerID="12869bcc05072358ac5908d16baf00763bf3ea4b8ca293b825ed486694afa02c"
Oct 30 13:15:20.644118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.644093    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889435-6fqwx"]
Oct 30 13:15:20.686378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.686347    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:20.808111 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.808077    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:20.829521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.829489    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ecd841-a5d5-43db-bb17-4b05e897b59b-kube-api-access-rlsr4\" (UniqueName: \"kubernetes.io/projected/02ecd841-a5d5-43db-bb17-4b05e897b59b-kube-api-access-rlsr4\") pod \"02ecd841-a5d5-43db-bb17-4b05e897b59b\" (UID: \"02ecd841-a5d5-43db-bb17-4b05e897b59b\") "
Oct 30 13:15:20.829702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.829601    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ecd841-a5d5-43db-bb17-4b05e897b59b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ecd841-a5d5-43db-bb17-4b05e897b59b-maas-api-service-ca\") pod \"02ecd841-a5d5-43db-bb17-4b05e897b59b\" (UID: \"02ecd841-a5d5-43db-bb17-4b05e897b59b\") "
Oct 30 13:15:20.830084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.830035    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ecd841-a5d5-43db-bb17-4b05e897b59b-maas-api-service-ca" pod "02ecd841-a5d5-43db-bb17-4b05e897b59b" (UID: "02ecd841-a5d5-43db-bb17-4b05e897b59b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.830579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.830416    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ecd841-a5d5-43db-bb17-4b05e897b59b-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.833272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.833202    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ecd841-a5d5-43db-bb17-4b05e897b59b-kube-api-access-rlsr4" pod "02ecd841-a5d5-43db-bb17-4b05e897b59b" (UID: "02ecd841-a5d5-43db-bb17-4b05e897b59b"). InnerVolumeSpecName "kube-api-access-rlsr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.888757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.888721    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"]
Oct 30 13:15:20.896402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.896369    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889435-6ktrh"]
Oct 30 13:15:20.931351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.931318    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-kube-api-access-vvbxx\" (UniqueName: \"kubernetes.io/projected/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-kube-api-access-vvbxx\") pod \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\" (UID: \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\") "
Oct 30 13:15:20.931538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.931372    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-maas-api-service-ca\") pod \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\" (UID: \"f1116de9-a0c9-4e5e-974f-ac6f8734f32d\") "
Oct 30 13:15:20.931809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.931778    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-maas-api-service-ca" pod "f1116de9-a0c9-4e5e-974f-ac6f8734f32d" (UID: "f1116de9-a0c9-4e5e-974f-ac6f8734f32d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.932127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.932101    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlsr4\" (UniqueName: \"kubernetes.io/projected/02ecd841-a5d5-43db-bb17-4b05e897b59b-kube-api-access-rlsr4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.932262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.932129    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.932262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.932156    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"]
Oct 30 13:15:20.933744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.933716    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-kube-api-access-vvbxx" pod "f1116de9-a0c9-4e5e-974f-ac6f8734f32d" (UID: "f1116de9-a0c9-4e5e-974f-ac6f8734f32d"). InnerVolumeSpecName "kube-api-access-vvbxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.936315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:20.936290    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889435-v94nq"]
Oct 30 13:15:21.040290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.034377    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvbxx\" (UniqueName: \"kubernetes.io/projected/f1116de9-a0c9-4e5e-974f-ac6f8734f32d-kube-api-access-vvbxx\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.451401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.451308    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:21.464419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.464390    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:21.588859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.588828    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"
Oct 30 13:15:21.591598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.591550    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g" event={"ID":"02ecd841-a5d5-43db-bb17-4b05e897b59b","Type":"ContainerDied","Data":"9dadb46a0075a9bea179352f44d578b8492921ec41b92aaa23e5d38644839571"}
Oct 30 13:15:21.591895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.591852    4993 scope.go:122] "RemoveContainer" containerID="08d9c97cbfba4a61a275e3e49a28843f68f168f30423d02f2286d34eba2b944c"
Oct 30 13:15:21.601381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.601352    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p" event={"ID":"f1116de9-a0c9-4e5e-974f-ac6f8734f32d","Type":"ContainerDied","Data":"82ac216994cccbb957d883375ccbb64d570e5154ec7f6a556546a972c18ac03a"}
Oct 30 13:15:21.601381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.601373    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"
Oct 30 13:15:21.618045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.616674    4993 scope.go:122] "RemoveContainer" containerID="a7cd004239fc97d399121c20b858e14859a6be77d7fc84d33445713fbb133825"
Oct 30 13:15:21.646316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.646185    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:15:21.659465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.659429    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"]
Oct 30 13:15:21.675608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.675557    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889435-2jg7g"]
Oct 30 13:15:21.698761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.698727    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" path="/var/lib/kubelet/pods/02ecd841-a5d5-43db-bb17-4b05e897b59b/volumes"
Oct 30 13:15:21.699940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.699870    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" path="/var/lib/kubelet/pods/2976e31a-dc21-49b8-b7b1-31084a53d7e3/volumes"
Oct 30 13:15:21.701379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.701355    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" path="/var/lib/kubelet/pods/50a59e6b-4b6c-4bf4-b5d6-86462b68e611/volumes"
Oct 30 13:15:21.702849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.702574    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" path="/var/lib/kubelet/pods/dc13c75b-d11f-4444-af69-5db6694deea3/volumes"
Oct 30 13:15:21.703688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.703575    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" path="/var/lib/kubelet/pods/fa7ea7d5-a0ae-4f82-b657-8f5869d6f759/volumes"
Oct 30 13:15:21.807952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.807914    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:21.862597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.862567    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:21.871726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.871697    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"]
Oct 30 13:15:21.876507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.876471    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ca097a-29889435-c5b7p"]
Oct 30 13:15:21.963045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.962931    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dcbb92-946a-4c0f-9627-9acc70f32934-maas-api-service-ca" pod "74dcbb92-946a-4c0f-9627-9acc70f32934" (UID: "74dcbb92-946a-4c0f-9627-9acc70f32934"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.963249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.963050    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74dcbb92-946a-4c0f-9627-9acc70f32934-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dcbb92-946a-4c0f-9627-9acc70f32934-maas-api-service-ca\") pod \"74dcbb92-946a-4c0f-9627-9acc70f32934\" (UID: \"74dcbb92-946a-4c0f-9627-9acc70f32934\") "
Oct 30 13:15:21.963249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.963111    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-maas-api-service-ca\") pod \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\" (UID: \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\") "
Oct 30 13:15:21.964166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.963194    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-kube-api-access-54khz\" (UniqueName: \"kubernetes.io/projected/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-kube-api-access-54khz\") pod \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\" (UID: \"bb18ce19-72f1-42ad-8301-b8c5903f9aaa\") "
Oct 30 13:15:21.964308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.964131    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-maas-api-service-ca" pod "bb18ce19-72f1-42ad-8301-b8c5903f9aaa" (UID: "bb18ce19-72f1-42ad-8301-b8c5903f9aaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.967260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.964906    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74dcbb92-946a-4c0f-9627-9acc70f32934-kube-api-access-qfqmt\" (UniqueName: \"kubernetes.io/projected/74dcbb92-946a-4c0f-9627-9acc70f32934-kube-api-access-qfqmt\") pod \"74dcbb92-946a-4c0f-9627-9acc70f32934\" (UID: \"74dcbb92-946a-4c0f-9627-9acc70f32934\") "
Oct 30 13:15:21.967969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.967942    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dcbb92-946a-4c0f-9627-9acc70f32934-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.968069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.967986    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.973824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.973721    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-kube-api-access-54khz" pod "bb18ce19-72f1-42ad-8301-b8c5903f9aaa" (UID: "bb18ce19-72f1-42ad-8301-b8c5903f9aaa"). InnerVolumeSpecName "kube-api-access-54khz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:21.974695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:21.974667    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dcbb92-946a-4c0f-9627-9acc70f32934-kube-api-access-qfqmt" pod "74dcbb92-946a-4c0f-9627-9acc70f32934" (UID: "74dcbb92-946a-4c0f-9627-9acc70f32934"). InnerVolumeSpecName "kube-api-access-qfqmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.069284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.069249    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54khz\" (UniqueName: \"kubernetes.io/projected/bb18ce19-72f1-42ad-8301-b8c5903f9aaa-kube-api-access-54khz\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.069284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.069284    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfqmt\" (UniqueName: \"kubernetes.io/projected/74dcbb92-946a-4c0f-9627-9acc70f32934-kube-api-access-qfqmt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.643697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.643655    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"
Oct 30 13:15:22.643697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.643665    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt" event={"ID":"bb18ce19-72f1-42ad-8301-b8c5903f9aaa","Type":"ContainerDied","Data":"2a993d9393aa755ffa931ba1d029f16a6ed8b92906feef01b631b2c10cb5d58b"}
Oct 30 13:15:22.644321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.643766    4993 scope.go:122] "RemoveContainer" containerID="b09aa8d55b547d26480e9a328a4eb818239a70d5f084c444bcedd0413deffad7"
Oct 30 13:15:22.646833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.646810    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"
Oct 30 13:15:22.646833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.646818    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm" event={"ID":"74dcbb92-946a-4c0f-9627-9acc70f32934","Type":"ContainerDied","Data":"b45c74780370a6ba0fa0ec89b2b3537519523d3a6173be73ad8d37d077402c2b"}
Oct 30 13:15:22.665350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.665327    4993 scope.go:122] "RemoveContainer" containerID="a86d2faeaf08f44e4e9797c6fddc82049ae2eb1b374f67098043f884642523ab"
Oct 30 13:15:22.714308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.714272    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"]
Oct 30 13:15:22.725760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.725725    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889435-tnqrt"]
Oct 30 13:15:22.765393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.765358    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"]
Oct 30 13:15:22.770008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:22.769980    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889435-xzkwm"]
Oct 30 13:15:23.692124 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:23.692092    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" path="/var/lib/kubelet/pods/74dcbb92-946a-4c0f-9627-9acc70f32934/volumes"
Oct 30 13:15:23.692612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:23.692597    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" path="/var/lib/kubelet/pods/bb18ce19-72f1-42ad-8301-b8c5903f9aaa/volumes"
Oct 30 13:15:23.693001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:23.692988    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" path="/var/lib/kubelet/pods/f1116de9-a0c9-4e5e-974f-ac6f8734f32d/volumes"
Oct 30 13:15:25.030573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:25.030530    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"]
Oct 30 13:15:25.042925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:25.042884    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a59fa51-cc4ch"]
Oct 30 13:15:25.693959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:25.693927    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4169f47f-be91-488c-a607-65a6253e3187" path="/var/lib/kubelet/pods/4169f47f-be91-488c-a607-65a6253e3187/volumes"
Oct 30 13:15:27.985251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.985196    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"]
Oct 30 13:15:27.985995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.985971    4993 cpu_manager.go:414] "containerMap: removing container" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.985997    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986010    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986017    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986033    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986041    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986056    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986063    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986075    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:15:27.986086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986082    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986092    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986098    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986111    4993 cpu_manager.go:414] "containerMap: removing container" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986118    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986134    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986141    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986154    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986161    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986173    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986179    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986189    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986196    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986225    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986233    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986247    4993 cpu_manager.go:414] "containerMap: removing container" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986255    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986264    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986272    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986283    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986290    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986305    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986312    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986325    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986332    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986345    4993 cpu_manager.go:414] "containerMap: removing container" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986353    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986368    4993 cpu_manager.go:414] "containerMap: removing container" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986376    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986391    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986399    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986413    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986420    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986441    4993 cpu_manager.go:414] "containerMap: removing container" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986449    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986460    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986467    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986479    4993 cpu_manager.go:414] "containerMap: removing container" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986487    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986501    4993 cpu_manager.go:414] "containerMap: removing container" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986509    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986524    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:15:27.986608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986532    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986696    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986712    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986722    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986731    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986740    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986749    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986759    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986766    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986776    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986784    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1116de9-a0c9-4e5e-974f-ac6f8734f32d" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986794    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986803    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986813    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986821    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986830    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986838    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986848    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ecd841-a5d5-43db-bb17-4b05e897b59b" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986901    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986913    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986923    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986932    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986940    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986949    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986959    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7ea7d5-a0ae-4f82-b657-8f5869d6f759" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986970    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:15:27.988629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.986981    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:15:27.990972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:27.990949    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.010348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.010302    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"]
Oct 30 13:15:28.017897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.017792    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:15:28.017897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.017933    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:15:28.018443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.017983    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:28.018443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.018040    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:15:28.057093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.057042    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.057296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.057192    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.057493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.057468    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnt2j\" (UniqueName: \"kubernetes.io/projected/23bda295-90f2-4710-b126-86a5271b5dc7-kube-api-access-dnt2j\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.158941 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.158885    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.159131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.159080    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt2j\" (UniqueName: \"kubernetes.io/projected/23bda295-90f2-4710-b126-86a5271b5dc7-kube-api-access-dnt2j\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.159183 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.159136    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.159825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.159797    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.161269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.161246    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.182158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.182123    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt2j\" (UniqueName: \"kubernetes.io/projected/23bda295-90f2-4710-b126-86a5271b5dc7-kube-api-access-dnt2j\") pod \"maas-api-revoke-sub-1f498539-7jt6n\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.306435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.306336    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:28.702702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.702674    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"]
Oct 30 13:15:28.705344 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:15:28.705312    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23bda295_90f2_4710_b126_86a5271b5dc7.slice/crio-574d091b5b6f94e5316be4e102d9d655094664008dd2a4d52fcc80f7fb5ac1d0 WatchSource:0}: Error finding container 574d091b5b6f94e5316be4e102d9d655094664008dd2a4d52fcc80f7fb5ac1d0: Status 404 returned error can't find the container with id 574d091b5b6f94e5316be4e102d9d655094664008dd2a4d52fcc80f7fb5ac1d0
Oct 30 13:15:28.723099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:28.723068    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n" event={"ID":"23bda295-90f2-4710-b126-86a5271b5dc7","Type":"ContainerStarted","Data":"574d091b5b6f94e5316be4e102d9d655094664008dd2a4d52fcc80f7fb5ac1d0"}
Oct 30 13:15:29.033441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:29.033404    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"]
Oct 30 13:15:29.037889 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:29.037857    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbec39d-n6wlp"]
Oct 30 13:15:29.694771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:29.694737    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb31659b-ab4b-44c7-b7ee-685f3115c220" path="/var/lib/kubelet/pods/fb31659b-ab4b-44c7-b7ee-685f3115c220/volumes"
Oct 30 13:15:29.733865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:29.733821    4993 generic.go:356] "Generic (PLEG): container finished" podID="23bda295-90f2-4710-b126-86a5271b5dc7" containerID="654916bf2be17a13827d02012f344a1f17154f3a885c95d1d968d264dc84ce86" exitCode=0
Oct 30 13:15:29.734031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:29.733911    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n" event={"ID":"23bda295-90f2-4710-b126-86a5271b5dc7","Type":"ContainerDied","Data":"654916bf2be17a13827d02012f344a1f17154f3a885c95d1d968d264dc84ce86"}
Oct 30 13:15:30.928934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:30.928905    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:31.022361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.022262    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-openshift-ca\") pod \"23bda295-90f2-4710-b126-86a5271b5dc7\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") "
Oct 30 13:15:31.022531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.022467    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23bda295-90f2-4710-b126-86a5271b5dc7-kube-api-access-dnt2j\" (UniqueName: \"kubernetes.io/projected/23bda295-90f2-4710-b126-86a5271b5dc7-kube-api-access-dnt2j\") pod \"23bda295-90f2-4710-b126-86a5271b5dc7\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") "
Oct 30 13:15:31.022589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.022535    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-xks-ca\") pod \"23bda295-90f2-4710-b126-86a5271b5dc7\" (UID: \"23bda295-90f2-4710-b126-86a5271b5dc7\") "
Oct 30 13:15:31.022695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.022664    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-openshift-ca" pod "23bda295-90f2-4710-b126-86a5271b5dc7" (UID: "23bda295-90f2-4710-b126-86a5271b5dc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:31.023180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.023156    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:31.025468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.025437    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-xks-ca" pod "23bda295-90f2-4710-b126-86a5271b5dc7" (UID: "23bda295-90f2-4710-b126-86a5271b5dc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:31.025620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.025464    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bda295-90f2-4710-b126-86a5271b5dc7-kube-api-access-dnt2j" pod "23bda295-90f2-4710-b126-86a5271b5dc7" (UID: "23bda295-90f2-4710-b126-86a5271b5dc7"). InnerVolumeSpecName "kube-api-access-dnt2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:31.124186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.124151    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnt2j\" (UniqueName: \"kubernetes.io/projected/23bda295-90f2-4710-b126-86a5271b5dc7-kube-api-access-dnt2j\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:31.124186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.124184    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bda295-90f2-4710-b126-86a5271b5dc7-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:15:31.759494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.759462    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"
Oct 30 13:15:31.759692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.759487    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n" event={"ID":"23bda295-90f2-4710-b126-86a5271b5dc7","Type":"ContainerDied","Data":"574d091b5b6f94e5316be4e102d9d655094664008dd2a4d52fcc80f7fb5ac1d0"}
Oct 30 13:15:31.759692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:31.759552    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574d091b5b6f94e5316be4e102d9d655094664008dd2a4d52fcc80f7fb5ac1d0"
Oct 30 13:15:38.578877 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:15:38.578826    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:15:45.053550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:45.053511    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"]
Oct 30 13:15:45.065947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:45.065894    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"]
Oct 30 13:15:45.070422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:45.070373    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-652e576f-7ngvm"]
Oct 30 13:15:45.074439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:45.074410    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab6a82f-6v79v"]
Oct 30 13:15:45.697644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:45.697601    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fca35ed-6494-4521-b6e9-32603df6c314" path="/var/lib/kubelet/pods/0fca35ed-6494-4521-b6e9-32603df6c314/volumes"
Oct 30 13:15:45.698443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:45.698416    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7d193e-0d98-4841-873b-afa134140a77" path="/var/lib/kubelet/pods/9e7d193e-0d98-4841-873b-afa134140a77/volumes"
Oct 30 13:15:46.037240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:46.037167    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"]
Oct 30 13:15:46.048943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:46.048906    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d904086b-qwnxc"]
Oct 30 13:15:47.693337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:47.693300    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb5cb66-de36-4304-99fd-33b2e746d161" path="/var/lib/kubelet/pods/ccb5cb66-de36-4304-99fd-33b2e746d161/volumes"
Oct 30 13:15:56.034194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:56.034100    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"]
Oct 30 13:15:56.044688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:56.044658    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8880e88-hllsd"]
Oct 30 13:15:57.694857 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:15:57.694818    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ad446f-37bf-4a6c-b477-3f58741cb7a8" path="/var/lib/kubelet/pods/a1ad446f-37bf-4a6c-b477-3f58741cb7a8/volumes"
Oct 30 13:16:01.038200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:01.038156    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"]
Oct 30 13:16:01.048622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:01.048587    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bd32e91-fqxvc"]
Oct 30 13:16:01.702335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:01.702295    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae6ca87-1e42-4c5d-b80b-8741d57892fa" path="/var/lib/kubelet/pods/4ae6ca87-1e42-4c5d-b80b-8741d57892fa/volumes"
Oct 30 13:16:03.031352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:03.031319    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"]
Oct 30 13:16:03.042894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:03.042863    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddb34a0-gjfgm"]
Oct 30 13:16:03.692799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:03.692762    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45bfe02-e73a-463c-ab19-5dba0c4fde09" path="/var/lib/kubelet/pods/b45bfe02-e73a-463c-ab19-5dba0c4fde09/volumes"
Oct 30 13:16:08.570803 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:16:08.570767    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:16:16.042288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:16.042248    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"]
Oct 30 13:16:16.053376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:16.053340    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c981bef-nfqv7"]
Oct 30 13:16:17.692640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:17.692605    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c57e62e-8eb3-4061-b01b-4114aa6c054c" path="/var/lib/kubelet/pods/6c57e62e-8eb3-4061-b01b-4114aa6c054c/volumes"
Oct 30 13:16:18.939996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:18.939957    4993 scope.go:122] "RemoveContainer" containerID="4ef97a13544cc67ca7719f3d94f3abff15ab88cb06d70b18160785bf0dbfc982"
Oct 30 13:16:18.961932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:18.961899    4993 scope.go:122] "RemoveContainer" containerID="276afc516d838b37c25baca8801e302f4e1dadb6942a109019f8a1898a2e91a8"
Oct 30 13:16:18.973490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:18.973461    4993 scope.go:122] "RemoveContainer" containerID="be8db447dead6de4a515a5ecbd2d211870027a77ff254039fb7e3170fd03f6cf"
Oct 30 13:16:18.985169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:18.985143    4993 scope.go:122] "RemoveContainer" containerID="17eacce0a4680dda7fcaa1a97806650cf471abe8548f0c7eca392d1eb4a4b77a"
Oct 30 13:16:18.998850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:18.998822    4993 scope.go:122] "RemoveContainer" containerID="704f91c55f9ada83d5aacc9b47f8a6e17fbc92ee4c27b7d4d33f2b429d5251ba"
Oct 30 13:16:19.010650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:19.010621    4993 scope.go:122] "RemoveContainer" containerID="25505c76838304350d610d34a16adbb1b4f5bab664debad37104cdad65e2a583"
Oct 30 13:16:19.021135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:19.021113    4993 scope.go:122] "RemoveContainer" containerID="bc256370767595db1d025782266054b089d26d572fbec1ad2da5faebc9f72f7f"
Oct 30 13:16:19.031922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:19.031895    4993 scope.go:122] "RemoveContainer" containerID="fec7fe0b4deb0a1d0be0c1e29f2f8c5067fbbc3dd565f5f29fab5e3a5f090760"
Oct 30 13:16:19.042485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:19.042455    4993 scope.go:122] "RemoveContainer" containerID="be7f293614bead2c03d5f0a1bb53edf342ec689f2621e4c43733909cca3812bc"
Oct 30 13:16:19.052591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:19.052564    4993 scope.go:122] "RemoveContainer" containerID="648752ec6e9c23dc18fe8e6fa47eb5910359f72abb687b3899bafb5e9023a03f"
Oct 30 13:16:19.063132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:19.063105    4993 scope.go:122] "RemoveContainer" containerID="d1c6acc20cfae7c03264e80b421bc41b9d32f77b8c26989398f3488d3652697f"
Oct 30 13:16:23.090905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:23.090854    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"]
Oct 30 13:16:23.102792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:23.102758    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffec38d-rtfqd"]
Oct 30 13:16:23.696289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:23.696249    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d05a65-b467-4da5-853b-9097504ab51e" path="/var/lib/kubelet/pods/b5d05a65-b467-4da5-853b-9097504ab51e/volumes"
Oct 30 13:16:37.051965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:37.051925    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"]
Oct 30 13:16:37.063629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:37.063597    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e914f582-hmxfk"]
Oct 30 13:16:37.692838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:37.692804    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3300c14c-ce34-4406-a8bc-059ec0180518" path="/var/lib/kubelet/pods/3300c14c-ce34-4406-a8bc-059ec0180518/volumes"
Oct 30 13:16:38.571152 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:16:38.571105    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:16:40.044379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:40.044342    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"]
Oct 30 13:16:40.056449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:40.056417    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3202a8e0-vqz9m"]
Oct 30 13:16:41.735668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:41.733586    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1" path="/var/lib/kubelet/pods/3c3bc684-7f56-4e8d-b4d0-45ad766f8ab1/volumes"
Oct 30 13:16:43.030829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:43.030792    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"]
Oct 30 13:16:43.044884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:43.044843    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"]
Oct 30 13:16:43.050767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:43.050722    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf4097-m26wr"]
Oct 30 13:16:43.056568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:43.056536    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5722bbb7-6xxcd"]
Oct 30 13:16:43.696025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:43.695985    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311b122f-7dbb-4986-8972-bb799bc94830" path="/var/lib/kubelet/pods/311b122f-7dbb-4986-8972-bb799bc94830/volumes"
Oct 30 13:16:43.696778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:43.696741    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e" path="/var/lib/kubelet/pods/43c75d8e-c134-48a8-9a4c-9e7a47a2bc1e/volumes"
Oct 30 13:16:49.693500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.693463    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"]
Oct 30 13:16:49.693999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.693984    4993 cpu_manager.go:414] "containerMap: removing container" podUID="23bda295-90f2-4710-b126-86a5271b5dc7" containerName="revoke-keys"
Oct 30 13:16:49.693999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.693999    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23bda295-90f2-4710-b126-86a5271b5dc7" containerName="revoke-keys"
Oct 30 13:16:49.694076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694017    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:16:49.694076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694022    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:16:49.694076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694042    4993 cpu_manager.go:414] "containerMap: removing container" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:16:49.694076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694047    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:16:49.694076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694053    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:16:49.694076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694058    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc13c75b-d11f-4444-af69-5db6694deea3" containerName="cleanup"
Oct 30 13:16:49.694076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694067    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:16:49.694076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694073    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694084    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694092    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694115    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694120    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694130    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694134    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694254    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="23bda295-90f2-4710-b126-86a5271b5dc7" containerName="revoke-keys"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694267    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3893102-450d-4e3d-974f-f57d67b662de" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694281    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c509bfa4-fa82-4fa1-ba3c-b8b4ced1b36a" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694291    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a59e6b-4b6c-4bf4-b5d6-86462b68e611" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694300    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce95e7b-befb-4079-a2ab-b83cfd7f741d" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694310    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dcbb92-946a-4c0f-9627-9acc70f32934" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694316    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb18ce19-72f1-42ad-8301-b8c5903f9aaa" containerName="cleanup"
Oct 30 13:16:49.694402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.694327    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2976e31a-dc21-49b8-b7b1-31084a53d7e3" containerName="cleanup"
Oct 30 13:16:49.697989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.697967    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.716172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.716142    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:16:49.717314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.717283    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"]
Oct 30 13:16:49.726317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.724530    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:16:49.769769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.769721    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:16:49.769769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.769727    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:16:49.833284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.833198    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5z6\" (UniqueName: \"kubernetes.io/projected/b0af9ad0-695c-4a4a-83e7-041d72627695-kube-api-access-ts5z6\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.833471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.833344    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.833585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.833568    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.934513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.934472    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.934724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.934577    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5z6\" (UniqueName: \"kubernetes.io/projected/b0af9ad0-695c-4a4a-83e7-041d72627695-kube-api-access-ts5z6\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.934724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.934618    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.935388 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.935358    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.937335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.937308    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:49.969967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:49.969875    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5z6\" (UniqueName: \"kubernetes.io/projected/b0af9ad0-695c-4a4a-83e7-041d72627695-kube-api-access-ts5z6\") pod \"maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:50.013364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:50.013325    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:50.373563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:50.373528    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"]
Oct 30 13:16:50.376678 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:16:50.376642    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0af9ad0_695c_4a4a_83e7_041d72627695.slice/crio-b05d096d968e06c58c06c7d5494338a7cf8f5e0655631e51e9ffeedf4142e707 WatchSource:0}: Error finding container b05d096d968e06c58c06c7d5494338a7cf8f5e0655631e51e9ffeedf4142e707: Status 404 returned error can't find the container with id b05d096d968e06c58c06c7d5494338a7cf8f5e0655631e51e9ffeedf4142e707
Oct 30 13:16:50.547875 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:50.547833    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb" event={"ID":"b0af9ad0-695c-4a4a-83e7-041d72627695","Type":"ContainerStarted","Data":"b05d096d968e06c58c06c7d5494338a7cf8f5e0655631e51e9ffeedf4142e707"}
Oct 30 13:16:51.559787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:51.559743    4993 generic.go:356] "Generic (PLEG): container finished" podID="b0af9ad0-695c-4a4a-83e7-041d72627695" containerID="65605793c67f5e4050bcec70fc903719ec266470e8b628c4a54122feb8b0db0c" exitCode=0
Oct 30 13:16:51.560358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:51.559874    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb" event={"ID":"b0af9ad0-695c-4a4a-83e7-041d72627695","Type":"ContainerDied","Data":"65605793c67f5e4050bcec70fc903719ec266470e8b628c4a54122feb8b0db0c"}
Oct 30 13:16:52.746387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.746340    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:52.891196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.891035    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-openshift-ca\") pod \"b0af9ad0-695c-4a4a-83e7-041d72627695\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") "
Oct 30 13:16:52.891196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.891131    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-xks-ca\") pod \"b0af9ad0-695c-4a4a-83e7-041d72627695\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") "
Oct 30 13:16:52.891496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.891263    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0af9ad0-695c-4a4a-83e7-041d72627695-kube-api-access-ts5z6\" (UniqueName: \"kubernetes.io/projected/b0af9ad0-695c-4a4a-83e7-041d72627695-kube-api-access-ts5z6\") pod \"b0af9ad0-695c-4a4a-83e7-041d72627695\" (UID: \"b0af9ad0-695c-4a4a-83e7-041d72627695\") "
Oct 30 13:16:52.891567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.891538    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-openshift-ca" pod "b0af9ad0-695c-4a4a-83e7-041d72627695" (UID: "b0af9ad0-695c-4a4a-83e7-041d72627695"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:52.892649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.892618    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:16:52.895632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.895593    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-xks-ca" pod "b0af9ad0-695c-4a4a-83e7-041d72627695" (UID: "b0af9ad0-695c-4a4a-83e7-041d72627695"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:52.895632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.895608    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0af9ad0-695c-4a4a-83e7-041d72627695-kube-api-access-ts5z6" pod "b0af9ad0-695c-4a4a-83e7-041d72627695" (UID: "b0af9ad0-695c-4a4a-83e7-041d72627695"). InnerVolumeSpecName "kube-api-access-ts5z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:52.993867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.993826    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0af9ad0-695c-4a4a-83e7-041d72627695-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:16:52.993867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:52.993867    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5z6\" (UniqueName: \"kubernetes.io/projected/b0af9ad0-695c-4a4a-83e7-041d72627695-kube-api-access-ts5z6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:16:53.580532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:53.580495    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"
Oct 30 13:16:53.580716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:53.580527    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb" event={"ID":"b0af9ad0-695c-4a4a-83e7-041d72627695","Type":"ContainerDied","Data":"b05d096d968e06c58c06c7d5494338a7cf8f5e0655631e51e9ffeedf4142e707"}
Oct 30 13:16:53.580716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:16:53.580577    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b05d096d968e06c58c06c7d5494338a7cf8f5e0655631e51e9ffeedf4142e707"
Oct 30 13:17:08.569225 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:17:08.569177    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:17:11.041667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:11.041621    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"]
Oct 30 13:17:11.054077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:11.054038    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e19a9414-wgd2p"]
Oct 30 13:17:11.693020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:11.692980    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b81dd24-27d8-46fc-90ad-f68d33d9e810" path="/var/lib/kubelet/pods/0b81dd24-27d8-46fc-90ad-f68d33d9e810/volumes"
Oct 30 13:17:16.163947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:16.163899    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:17:16.178969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:16.178935    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:17:19.217829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.217792    4993 scope.go:122] "RemoveContainer" containerID="5cf05a1cad6e3a77184141089c6a3f9054030f5cdeaf4bf6261b7a72bd839541"
Oct 30 13:17:19.228297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.228257    4993 scope.go:122] "RemoveContainer" containerID="d09842d03b9928a6ae1c6b2306951171d49b619b5ea78cf85c3833bfa5a292b2"
Oct 30 13:17:19.237601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.237579    4993 scope.go:122] "RemoveContainer" containerID="ab2f97cd490f77924297c1c4d27454c219206320473f7c413f0f25b6bea1da60"
Oct 30 13:17:19.246699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.246679    4993 scope.go:122] "RemoveContainer" containerID="40a75284fff08459de10720c0e2cd4f02e3302d71d9febdf12b931fd2cc0401a"
Oct 30 13:17:19.256857 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.256838    4993 scope.go:122] "RemoveContainer" containerID="cdb404d24849353872c7f10799bbce8aa04c8ff5afec853bf31d0747b4654548"
Oct 30 13:17:19.265947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.265922    4993 scope.go:122] "RemoveContainer" containerID="1fe58ce27797e5890905e98cd761e658fb572abf3c231faa1619d341926cbfd4"
Oct 30 13:17:19.275018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.274995    4993 scope.go:122] "RemoveContainer" containerID="91312ebbeb4ed96f9eedabb0602644cb3a38f27532dd607a8b5366a7172fee97"
Oct 30 13:17:19.284558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.284538    4993 scope.go:122] "RemoveContainer" containerID="08f7a8ab4ac414e2e11d0e89ad21f39bb7b5f6608fe253a61ea08d31216386a3"
Oct 30 13:17:19.295872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.295847    4993 scope.go:122] "RemoveContainer" containerID="5a1a7400a28c7e85170903c28b7063223eef83ee741e0d9d6196ae64c7c4f36a"
Oct 30 13:17:19.304825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.304799    4993 scope.go:122] "RemoveContainer" containerID="0819c17c7fe635e4f4138c604bc0a6bfe802429fd1bc5492acc90f1f7d72414a"
Oct 30 13:17:19.313674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.313650    4993 scope.go:122] "RemoveContainer" containerID="6b03e9f8d7efd9a07dfc3b409e685bc9162384314788ff236d9d02e8003dd346"
Oct 30 13:17:19.322460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:19.322441    4993 scope.go:122] "RemoveContainer" containerID="c87d275b517e55813b3e595456a9c00199b3522a5bdac15d7191d5acdf5fdafb"
Oct 30 13:17:23.974059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:23.974000    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"]
Oct 30 13:17:23.988504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:23.988454    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ca097a-6zsbb"]
Oct 30 13:17:25.693184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:25.693149    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0af9ad0-695c-4a4a-83e7-041d72627695" path="/var/lib/kubelet/pods/b0af9ad0-695c-4a4a-83e7-041d72627695/volumes"
Oct 30 13:17:28.697297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.697246    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"]
Oct 30 13:17:28.698084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.698062    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b0af9ad0-695c-4a4a-83e7-041d72627695" containerName="revoke-keys"
Oct 30 13:17:28.698130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.698088    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0af9ad0-695c-4a4a-83e7-041d72627695" containerName="revoke-keys"
Oct 30 13:17:28.698332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.698318    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0af9ad0-695c-4a4a-83e7-041d72627695" containerName="revoke-keys"
Oct 30 13:17:28.703902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.703873    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.719742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.719713    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"]
Oct 30 13:17:28.727732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.727698    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:28.728416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.728387    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:17:28.728559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.728468    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:28.730134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.730107    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:28.764115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.764005    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774cm\" (UniqueName: \"kubernetes.io/projected/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-kube-api-access-774cm\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.764824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.764692    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.764824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.764792    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.866829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.866786    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.867007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.866842    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.867007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.866916    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774cm\" (UniqueName: \"kubernetes.io/projected/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-kube-api-access-774cm\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.867682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.867660    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.869347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.869324    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:28.890917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:28.890887    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774cm\" (UniqueName: \"kubernetes.io/projected/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-kube-api-access-774cm\") pod \"maas-api-revoke-sub-38547e1c-4scdd\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:29.019325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:29.019291    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:29.339352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:29.339322    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"]
Oct 30 13:17:29.342140 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:17:29.342106    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc425e1cc_f94d_4b39_a1c6_6e6aa52e94c1.slice/crio-4cebf46fc0c30fdae9f98b1e14271347c73c73912218a55d6227a7270f405c65 WatchSource:0}: Error finding container 4cebf46fc0c30fdae9f98b1e14271347c73c73912218a55d6227a7270f405c65: Status 404 returned error can't find the container with id 4cebf46fc0c30fdae9f98b1e14271347c73c73912218a55d6227a7270f405c65
Oct 30 13:17:29.931644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:29.931546    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd" event={"ID":"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1","Type":"ContainerStarted","Data":"26951efafc19af710f7ad4b54f173ddbbaecdd4e071c8b91895fe09e1c3adb45"}
Oct 30 13:17:29.931644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:29.931608    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd" event={"ID":"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1","Type":"ContainerStarted","Data":"4cebf46fc0c30fdae9f98b1e14271347c73c73912218a55d6227a7270f405c65"}
Oct 30 13:17:29.975298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:29.975240    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd" podStartSLOduration=1.975221248 podStartE2EDuration="1.975221248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:29.974096951 +0000 UTC m=+2714.449816562" watchObservedRunningTime="2026-10-30 13:17:29.975221248 +0000 UTC m=+2714.450940856"
Oct 30 13:17:30.941261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:30.941222    4993 generic.go:356] "Generic (PLEG): container finished" podID="c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1" containerID="26951efafc19af710f7ad4b54f173ddbbaecdd4e071c8b91895fe09e1c3adb45" exitCode=0
Oct 30 13:17:30.941753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:30.941305    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd" event={"ID":"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1","Type":"ContainerDied","Data":"26951efafc19af710f7ad4b54f173ddbbaecdd4e071c8b91895fe09e1c3adb45"}
Oct 30 13:17:32.142295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.142268    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:32.216079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.215983    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-kube-api-access-774cm\" (UniqueName: \"kubernetes.io/projected/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-kube-api-access-774cm\") pod \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") "
Oct 30 13:17:32.216291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.216199    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-xks-ca\") pod \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") "
Oct 30 13:17:32.216417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.216400    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-openshift-ca\") pod \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\" (UID: \"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1\") "
Oct 30 13:17:32.216829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.216803    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-openshift-ca" pod "c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1" (UID: "c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:32.216996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.216979    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:32.218671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.218634    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-kube-api-access-774cm" pod "c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1" (UID: "c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1"). InnerVolumeSpecName "kube-api-access-774cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:32.219345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.219315    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-xks-ca" pod "c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1" (UID: "c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:32.318364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.318319    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:32.318364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.318358    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-774cm\" (UniqueName: \"kubernetes.io/projected/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1-kube-api-access-774cm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:32.957486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.957445    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd" event={"ID":"c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1","Type":"ContainerDied","Data":"4cebf46fc0c30fdae9f98b1e14271347c73c73912218a55d6227a7270f405c65"}
Oct 30 13:17:32.957681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.957511    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cebf46fc0c30fdae9f98b1e14271347c73c73912218a55d6227a7270f405c65"
Oct 30 13:17:32.957681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:32.957462    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"
Oct 30 13:17:38.584014 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:17:38.583969    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:17:39.171088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.171038    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"]
Oct 30 13:17:39.171677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.171657    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1" containerName="revoke-keys"
Oct 30 13:17:39.171677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.171676    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1" containerName="revoke-keys"
Oct 30 13:17:39.171849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.171768    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1" containerName="revoke-keys"
Oct 30 13:17:39.175158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.175136    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.190016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.189980    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:39.190167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.190014    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:39.206599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.206564    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"]
Oct 30 13:17:39.224222 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.224174    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:17:39.224427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.224302    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:39.245037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.244995    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.245418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.245385    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.245508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.245443    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72lg\" (UniqueName: \"kubernetes.io/projected/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-kube-api-access-c72lg\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.347174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.347126    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.347430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.347408    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.347515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.347453    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72lg\" (UniqueName: \"kubernetes.io/projected/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-kube-api-access-c72lg\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.348053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.348017    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.350080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.350053    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.371503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.371454    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72lg\" (UniqueName: \"kubernetes.io/projected/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-kube-api-access-c72lg\") pod \"maas-api-revoke-sub-b231f234-v6t57\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.491890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.491853    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:39.818796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:39.818712    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"]
Oct 30 13:17:39.822987 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:17:39.822947    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d93c23d_75fc_432e_bf15_0eab1b8ef2c7.slice/crio-3fc17f3465bdc06743d78099a236a41ea03e44e8a40e19e3e37d676d4e9794e6 WatchSource:0}: Error finding container 3fc17f3465bdc06743d78099a236a41ea03e44e8a40e19e3e37d676d4e9794e6: Status 404 returned error can't find the container with id 3fc17f3465bdc06743d78099a236a41ea03e44e8a40e19e3e37d676d4e9794e6
Oct 30 13:17:40.021182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:40.021145    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57" event={"ID":"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7","Type":"ContainerStarted","Data":"3fc17f3465bdc06743d78099a236a41ea03e44e8a40e19e3e37d676d4e9794e6"}
Oct 30 13:17:41.031141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:41.031100    4993 generic.go:356] "Generic (PLEG): container finished" podID="5d93c23d-75fc-432e-bf15-0eab1b8ef2c7" containerID="3f0523bfb167385bf6db3b6a056ecbc7b2a7b58a43bcdc2ae6a0321490bf5654" exitCode=0
Oct 30 13:17:41.031567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:41.031184    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57" event={"ID":"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7","Type":"ContainerDied","Data":"3f0523bfb167385bf6db3b6a056ecbc7b2a7b58a43bcdc2ae6a0321490bf5654"}
Oct 30 13:17:42.223112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.223085    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:42.334181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.334056    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-openshift-ca\") pod \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") "
Oct 30 13:17:42.334181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.334133    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-kube-api-access-c72lg\" (UniqueName: \"kubernetes.io/projected/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-kube-api-access-c72lg\") pod \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") "
Oct 30 13:17:42.334462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.334182    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-xks-ca\") pod \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\" (UID: \"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7\") "
Oct 30 13:17:42.334629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.334593    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-openshift-ca" pod "5d93c23d-75fc-432e-bf15-0eab1b8ef2c7" (UID: "5d93c23d-75fc-432e-bf15-0eab1b8ef2c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:42.335067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.335038    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:42.338068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.338034    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-kube-api-access-c72lg" pod "5d93c23d-75fc-432e-bf15-0eab1b8ef2c7" (UID: "5d93c23d-75fc-432e-bf15-0eab1b8ef2c7"). InnerVolumeSpecName "kube-api-access-c72lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:42.342626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.342580    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-xks-ca" pod "5d93c23d-75fc-432e-bf15-0eab1b8ef2c7" (UID: "5d93c23d-75fc-432e-bf15-0eab1b8ef2c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:42.436813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.436763    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72lg\" (UniqueName: \"kubernetes.io/projected/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-kube-api-access-c72lg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:42.436813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:42.436811    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:43.058391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:43.058345    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"
Oct 30 13:17:43.058391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:43.058357    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57" event={"ID":"5d93c23d-75fc-432e-bf15-0eab1b8ef2c7","Type":"ContainerDied","Data":"3fc17f3465bdc06743d78099a236a41ea03e44e8a40e19e3e37d676d4e9794e6"}
Oct 30 13:17:43.058637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:43.058416    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc17f3465bdc06743d78099a236a41ea03e44e8a40e19e3e37d676d4e9794e6"
Oct 30 13:17:50.922266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.922221    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"]
Oct 30 13:17:50.923390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.923362    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5d93c23d-75fc-432e-bf15-0eab1b8ef2c7" containerName="revoke-keys"
Oct 30 13:17:50.923390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.923388    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d93c23d-75fc-432e-bf15-0eab1b8ef2c7" containerName="revoke-keys"
Oct 30 13:17:50.923599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.923568    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d93c23d-75fc-432e-bf15-0eab1b8ef2c7" containerName="revoke-keys"
Oct 30 13:17:50.926964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.926942    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:50.938025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.935886    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"]
Oct 30 13:17:50.943310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.943278    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:50.943310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.943281    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:50.943551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.943460    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:50.953508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:50.953466    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:17:51.057535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.057490    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.057747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.057543    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.057867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.057846    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr49n\" (UniqueName: \"kubernetes.io/projected/3942337b-3aad-46dc-a969-2390b55cea83-kube-api-access-zr49n\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.159057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.159020    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.159284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.159068    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.159365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.159344    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr49n\" (UniqueName: \"kubernetes.io/projected/3942337b-3aad-46dc-a969-2390b55cea83-kube-api-access-zr49n\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.160034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.160000    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.161928 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.161901    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.183094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.183009    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr49n\" (UniqueName: \"kubernetes.io/projected/3942337b-3aad-46dc-a969-2390b55cea83-kube-api-access-zr49n\") pod \"maas-api-revoke-sub-e1ae6f37-zh5rr\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.242888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.242854    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:51.363530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.363472    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"]
Oct 30 13:17:51.634496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:51.633534    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"]
Oct 30 13:17:51.634496 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:17:51.634340    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3942337b_3aad_46dc_a969_2390b55cea83.slice/crio-c06efe5673f7f9d379f40010726670c5aac817e28db844d4915cf5dd97d457b1 WatchSource:0}: Error finding container c06efe5673f7f9d379f40010726670c5aac817e28db844d4915cf5dd97d457b1: Status 404 returned error can't find the container with id c06efe5673f7f9d379f40010726670c5aac817e28db844d4915cf5dd97d457b1
Oct 30 13:17:52.247163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:52.247123    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr" event={"ID":"3942337b-3aad-46dc-a969-2390b55cea83","Type":"ContainerStarted","Data":"f34ad4abdf4684b6b44de82674ecfd95e73fe18916aeeec4f3085764924ce953"}
Oct 30 13:17:52.247711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:52.247193    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr" event={"ID":"3942337b-3aad-46dc-a969-2390b55cea83","Type":"ContainerStarted","Data":"c06efe5673f7f9d379f40010726670c5aac817e28db844d4915cf5dd97d457b1"}
Oct 30 13:17:52.290728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:52.290664    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr" podStartSLOduration=2.290646229 podStartE2EDuration="2.290646229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:52.286667268 +0000 UTC m=+2736.762386877" watchObservedRunningTime="2026-10-30 13:17:52.290646229 +0000 UTC m=+2736.766365832"
Oct 30 13:17:53.255974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:53.255932    4993 generic.go:356] "Generic (PLEG): container finished" podID="3942337b-3aad-46dc-a969-2390b55cea83" containerID="f34ad4abdf4684b6b44de82674ecfd95e73fe18916aeeec4f3085764924ce953" exitCode=0
Oct 30 13:17:53.256525 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:53.255996    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr" event={"ID":"3942337b-3aad-46dc-a969-2390b55cea83","Type":"ContainerDied","Data":"f34ad4abdf4684b6b44de82674ecfd95e73fe18916aeeec4f3085764924ce953"}
Oct 30 13:17:54.489713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.489684    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:54.629412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.629320    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3942337b-3aad-46dc-a969-2390b55cea83-kube-api-access-zr49n\" (UniqueName: \"kubernetes.io/projected/3942337b-3aad-46dc-a969-2390b55cea83-kube-api-access-zr49n\") pod \"3942337b-3aad-46dc-a969-2390b55cea83\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") "
Oct 30 13:17:54.629588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.629553    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-xks-ca\") pod \"3942337b-3aad-46dc-a969-2390b55cea83\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") "
Oct 30 13:17:54.629709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.629692    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-openshift-ca\") pod \"3942337b-3aad-46dc-a969-2390b55cea83\" (UID: \"3942337b-3aad-46dc-a969-2390b55cea83\") "
Oct 30 13:17:54.630060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.630032    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-openshift-ca" pod "3942337b-3aad-46dc-a969-2390b55cea83" (UID: "3942337b-3aad-46dc-a969-2390b55cea83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:54.630273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.630255    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:54.631713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.631685    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3942337b-3aad-46dc-a969-2390b55cea83-kube-api-access-zr49n" pod "3942337b-3aad-46dc-a969-2390b55cea83" (UID: "3942337b-3aad-46dc-a969-2390b55cea83"). InnerVolumeSpecName "kube-api-access-zr49n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:54.632013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.631991    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-xks-ca" pod "3942337b-3aad-46dc-a969-2390b55cea83" (UID: "3942337b-3aad-46dc-a969-2390b55cea83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:54.731296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.731249    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3942337b-3aad-46dc-a969-2390b55cea83-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:54.731296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:54.731284    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr49n\" (UniqueName: \"kubernetes.io/projected/3942337b-3aad-46dc-a969-2390b55cea83-kube-api-access-zr49n\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:17:55.278363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:55.278327    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"
Oct 30 13:17:55.278363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:55.278345    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr" event={"ID":"3942337b-3aad-46dc-a969-2390b55cea83","Type":"ContainerDied","Data":"c06efe5673f7f9d379f40010726670c5aac817e28db844d4915cf5dd97d457b1"}
Oct 30 13:17:55.278571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:55.278410    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c06efe5673f7f9d379f40010726670c5aac817e28db844d4915cf5dd97d457b1"
Oct 30 13:17:55.723258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:55.723161    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"]
Oct 30 13:17:55.723668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:17:55.723381    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" containerID="cri-o://ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb" gracePeriod=30
Oct 30 13:18:00.126992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:00.126904    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qtjv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:18:00.127522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:00.126986    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:18:00.824150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:00.824105    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"]
Oct 30 13:18:01.132977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.132948    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 13:18:01.153886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.153853    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-certs\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.153918    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-credential-socket\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.153969    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-podinfo\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154038    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-kube-api-access-h942k\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-kube-api-access-h942k\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154101    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-envoy\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154141    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-token\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-token\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154194    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-credential-socket" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.154325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154253    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-certs" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.154325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154290    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-data\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154328    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-socket\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154378    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istiod-ca-cert\") pod \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\" (UID: \"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5\") "
Oct 30 13:18:01.154628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154648    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-data" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.155018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.154992    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istiod-ca-cert" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:01.155307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.155129    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.155418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.155326    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.155418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.155344    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.155418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.155090    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-socket" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.156819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.156792    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-token" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:01.156938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.156816    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-envoy" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.157135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.157116    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-podinfo" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:18:01.157337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.157316    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-kube-api-access-h942k" pod "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" (UID: "1bf5adae-c15d-4e5e-8c24-0edbebff5ef5"). InnerVolumeSpecName "kube-api-access-h942k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:01.256798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.256763    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.256798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.256796    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h942k\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-kube-api-access-h942k\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.256798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.256807    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.257040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.256815    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.257040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.256825    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.257040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.256832    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.369630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.369513    4993 generic.go:356] "Generic (PLEG): container finished" podID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerID="ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb" exitCode=0
Oct 30 13:18:01.369976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.369638    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"
Oct 30 13:18:01.370621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.369661    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" event={"ID":"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5","Type":"ContainerDied","Data":"ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb"}
Oct 30 13:18:01.370621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.370144    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7" event={"ID":"1bf5adae-c15d-4e5e-8c24-0edbebff5ef5","Type":"ContainerDied","Data":"491bba2a8967de2c48c1709b15e8f4e5a90fa91f6522a407c247d9efe4684093"}
Oct 30 13:18:01.370621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.370187    4993 scope.go:122] "RemoveContainer" containerID="ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb"
Oct 30 13:18:01.390051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.390026    4993 scope.go:122] "RemoveContainer" containerID="ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb"
Oct 30 13:18:01.390486 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:18:01.390458    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb\": container with ID starting with ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb not found: ID does not exist" containerID="ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb"
Oct 30 13:18:01.390595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.390496    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb"} err="failed to get container status \"ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb\": rpc error: code = NotFound desc = could not find container \"ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb\": container with ID starting with ec52ce9ca0d6cec37970bd7f5fa5a2376673dff51f272c80cb84a2705eb25cdb not found: ID does not exist"
Oct 30 13:18:01.436616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.436580    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"]
Oct 30 13:18:01.443036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.443003    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qtjv7"]
Oct 30 13:18:01.474146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.474106    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"]
Oct 30 13:18:01.474695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.474681    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3942337b-3aad-46dc-a969-2390b55cea83" containerName="revoke-keys"
Oct 30 13:18:01.474748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.474697    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3942337b-3aad-46dc-a969-2390b55cea83" containerName="revoke-keys"
Oct 30 13:18:01.474748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.474722    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy"
Oct 30 13:18:01.474748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.474727    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy"
Oct 30 13:18:01.474847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.474819    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3942337b-3aad-46dc-a969-2390b55cea83" containerName="revoke-keys"
Oct 30 13:18:01.474847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.474834    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" containerName="istio-proxy"
Oct 30 13:18:01.479831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.479808    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.495377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.495341    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:18:01.499271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.498304    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:18:01.499271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.498422    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:18:01.501261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.501237    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"]
Oct 30 13:18:01.562252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.562169    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kq9p\" (UniqueName: \"kubernetes.io/projected/c661b0b0-3112-4103-8726-a9c393b027e2-kube-api-access-6kq9p\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.562445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.562255    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.562445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.562382    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.596050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.596015    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:18:01.663572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.663487    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.663732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.663573    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.663732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.663714    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kq9p\" (UniqueName: \"kubernetes.io/projected/c661b0b0-3112-4103-8726-a9c393b027e2-kube-api-access-6kq9p\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.664284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.664257    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.665895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.665873    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.689330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.689301    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kq9p\" (UniqueName: \"kubernetes.io/projected/c661b0b0-3112-4103-8726-a9c393b027e2-kube-api-access-6kq9p\") pod \"maas-api-revoke-sub-ec5ae558-vjhtz\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:01.692713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.692682    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf5adae-c15d-4e5e-8c24-0edbebff5ef5" path="/var/lib/kubelet/pods/1bf5adae-c15d-4e5e-8c24-0edbebff5ef5/volumes"
Oct 30 13:18:01.794434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:01.794400    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:02.119239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:02.118779    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"]
Oct 30 13:18:02.124090 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:18:02.124054    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc661b0b0_3112_4103_8726_a9c393b027e2.slice/crio-fc0c8ddf495a4e00ccd55ab04790d89423fab2da0e5d7c8e575323e09381cf3e WatchSource:0}: Error finding container fc0c8ddf495a4e00ccd55ab04790d89423fab2da0e5d7c8e575323e09381cf3e: Status 404 returned error can't find the container with id fc0c8ddf495a4e00ccd55ab04790d89423fab2da0e5d7c8e575323e09381cf3e
Oct 30 13:18:02.411914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:02.411814    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz" event={"ID":"c661b0b0-3112-4103-8726-a9c393b027e2","Type":"ContainerStarted","Data":"fc0c8ddf495a4e00ccd55ab04790d89423fab2da0e5d7c8e575323e09381cf3e"}
Oct 30 13:18:03.434970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:03.434924    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz" event={"ID":"c661b0b0-3112-4103-8726-a9c393b027e2","Type":"ContainerStarted","Data":"b7565e303ed717477edebcb9a825682572df9c40db2bb4e0b329603d78167d6c"}
Oct 30 13:18:03.480021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:03.479968    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz" podStartSLOduration=2.479953686 podStartE2EDuration="2.479953686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:03.47571899 +0000 UTC m=+2747.951438612" watchObservedRunningTime="2026-10-30 13:18:03.479953686 +0000 UTC m=+2747.955673296"
Oct 30 13:18:04.447072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:04.447035    4993 generic.go:356] "Generic (PLEG): container finished" podID="c661b0b0-3112-4103-8726-a9c393b027e2" containerID="b7565e303ed717477edebcb9a825682572df9c40db2bb4e0b329603d78167d6c" exitCode=0
Oct 30 13:18:04.447635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:04.447098    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz" event={"ID":"c661b0b0-3112-4103-8726-a9c393b027e2","Type":"ContainerDied","Data":"b7565e303ed717477edebcb9a825682572df9c40db2bb4e0b329603d78167d6c"}
Oct 30 13:18:05.638731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.638701    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:05.752873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.752840    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-openshift-ca\") pod \"c661b0b0-3112-4103-8726-a9c393b027e2\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") "
Oct 30 13:18:05.753083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.752951    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c661b0b0-3112-4103-8726-a9c393b027e2-kube-api-access-6kq9p\" (UniqueName: \"kubernetes.io/projected/c661b0b0-3112-4103-8726-a9c393b027e2-kube-api-access-6kq9p\") pod \"c661b0b0-3112-4103-8726-a9c393b027e2\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") "
Oct 30 13:18:05.753083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.753040    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-xks-ca\") pod \"c661b0b0-3112-4103-8726-a9c393b027e2\" (UID: \"c661b0b0-3112-4103-8726-a9c393b027e2\") "
Oct 30 13:18:05.753472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.753309    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-openshift-ca" pod "c661b0b0-3112-4103-8726-a9c393b027e2" (UID: "c661b0b0-3112-4103-8726-a9c393b027e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:05.753708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.753683    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:05.755604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.755573    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c661b0b0-3112-4103-8726-a9c393b027e2-kube-api-access-6kq9p" pod "c661b0b0-3112-4103-8726-a9c393b027e2" (UID: "c661b0b0-3112-4103-8726-a9c393b027e2"). InnerVolumeSpecName "kube-api-access-6kq9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:05.755771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.755747    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-xks-ca" pod "c661b0b0-3112-4103-8726-a9c393b027e2" (UID: "c661b0b0-3112-4103-8726-a9c393b027e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:05.855081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.855037    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kq9p\" (UniqueName: \"kubernetes.io/projected/c661b0b0-3112-4103-8726-a9c393b027e2-kube-api-access-6kq9p\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:05.855081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:05.855073    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c661b0b0-3112-4103-8726-a9c393b027e2-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:06.400738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.400667    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf" podUID="4578924e-2c82-4845-a56c-149ac69bfd21" containerName="main" containerID="cri-o://25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269" gracePeriod=45
Oct 30 13:18:06.463997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.463966    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"
Oct 30 13:18:06.464157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.463961    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz" event={"ID":"c661b0b0-3112-4103-8726-a9c393b027e2","Type":"ContainerDied","Data":"fc0c8ddf495a4e00ccd55ab04790d89423fab2da0e5d7c8e575323e09381cf3e"}
Oct 30 13:18:06.464157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.464123    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0c8ddf495a4e00ccd55ab04790d89423fab2da0e5d7c8e575323e09381cf3e"
Oct 30 13:18:06.716887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.716859    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:18:06.874572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.874537    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-dshm\") pod \"4578924e-2c82-4845-a56c-149ac69bfd21\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") "
Oct 30 13:18:06.874770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.874593    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-model-cache\") pod \"4578924e-2c82-4845-a56c-149ac69bfd21\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") "
Oct 30 13:18:06.874770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.874643    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-home\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-home\") pod \"4578924e-2c82-4845-a56c-149ac69bfd21\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") "
Oct 30 13:18:06.874770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.874678    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4578924e-2c82-4845-a56c-149ac69bfd21-kube-api-access-nmcn8\" (UniqueName: \"kubernetes.io/projected/4578924e-2c82-4845-a56c-149ac69bfd21-kube-api-access-nmcn8\") pod \"4578924e-2c82-4845-a56c-149ac69bfd21\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") "
Oct 30 13:18:06.874944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.874914    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-model-cache" pod "4578924e-2c82-4845-a56c-149ac69bfd21" (UID: "4578924e-2c82-4845-a56c-149ac69bfd21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:06.875002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.874921    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-tmp-dir\") pod \"4578924e-2c82-4845-a56c-149ac69bfd21\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") "
Oct 30 13:18:06.875115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.875094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4578924e-2c82-4845-a56c-149ac69bfd21-tls-certs\" (UniqueName: \"kubernetes.io/secret/4578924e-2c82-4845-a56c-149ac69bfd21-tls-certs\") pod \"4578924e-2c82-4845-a56c-149ac69bfd21\" (UID: \"4578924e-2c82-4845-a56c-149ac69bfd21\") "
Oct 30 13:18:06.875388 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.875362    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-home" pod "4578924e-2c82-4845-a56c-149ac69bfd21" (UID: "4578924e-2c82-4845-a56c-149ac69bfd21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:06.875770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.875486    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-tmp-dir" pod "4578924e-2c82-4845-a56c-149ac69bfd21" (UID: "4578924e-2c82-4845-a56c-149ac69bfd21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:06.876570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.875827    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:06.876570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.875851    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:06.876570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.875863    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:06.878004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.877967    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4578924e-2c82-4845-a56c-149ac69bfd21-tls-certs" pod "4578924e-2c82-4845-a56c-149ac69bfd21" (UID: "4578924e-2c82-4845-a56c-149ac69bfd21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:06.879914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.879892    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-dshm" pod "4578924e-2c82-4845-a56c-149ac69bfd21" (UID: "4578924e-2c82-4845-a56c-149ac69bfd21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:06.881265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.881246    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4578924e-2c82-4845-a56c-149ac69bfd21-kube-api-access-nmcn8" pod "4578924e-2c82-4845-a56c-149ac69bfd21" (UID: "4578924e-2c82-4845-a56c-149ac69bfd21"). InnerVolumeSpecName "kube-api-access-nmcn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:06.977770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.977730    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4578924e-2c82-4845-a56c-149ac69bfd21-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:06.977770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.977765    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4578924e-2c82-4845-a56c-149ac69bfd21-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:06.977985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:06.977779    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmcn8\" (UniqueName: \"kubernetes.io/projected/4578924e-2c82-4845-a56c-149ac69bfd21-kube-api-access-nmcn8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:07.473996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.473906    4993 generic.go:356] "Generic (PLEG): container finished" podID="4578924e-2c82-4845-a56c-149ac69bfd21" containerID="25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269" exitCode=0
Oct 30 13:18:07.473996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.473969    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"
Oct 30 13:18:07.474279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.474021    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf" event={"ID":"4578924e-2c82-4845-a56c-149ac69bfd21","Type":"ContainerDied","Data":"25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269"}
Oct 30 13:18:07.474279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.474054    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf" event={"ID":"4578924e-2c82-4845-a56c-149ac69bfd21","Type":"ContainerDied","Data":"b64c1e9e2699bb03e4ef6d19407064f51286371653a2035e60860b8c03c15e93"}
Oct 30 13:18:07.474279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.474069    4993 scope.go:122] "RemoveContainer" containerID="25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269"
Oct 30 13:18:07.502418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.502385    4993 scope.go:122] "RemoveContainer" containerID="25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269"
Oct 30 13:18:07.502836 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:18:07.502796    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269\": container with ID starting with 25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269 not found: ID does not exist" containerID="25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269"
Oct 30 13:18:07.502933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.502852    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269"} err="failed to get container status \"25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269\": rpc error: code = NotFound desc = could not find container \"25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269\": container with ID starting with 25817a3cfcccd2e58678cf94831d5c5f4f0d6d870c93def0607ed014efaa9269 not found: ID does not exist"
Oct 30 13:18:07.537079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.537040    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"]
Oct 30 13:18:07.549129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.549088    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/auth-test-model-e9cb4a-kserve-7c78668469-hnggf"]
Oct 30 13:18:07.695080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:07.695042    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4578924e-2c82-4845-a56c-149ac69bfd21" path="/var/lib/kubelet/pods/4578924e-2c82-4845-a56c-149ac69bfd21/volumes"
Oct 30 13:18:08.570707 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:18:08.570669    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:18:12.904714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:12.904674    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"]
Oct 30 13:18:12.905182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:12.904914    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy" containerID="cri-o://7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd" gracePeriod=30
Oct 30 13:18:15.846480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:15.846416    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs" podUID="2838ebe2-712b-4d88-881d-c4c17c6308f6" containerName="main" containerID="cri-o://6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4" gracePeriod=45
Oct 30 13:18:16.154649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.154507    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:18:16.306659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.306619    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-model-cache\") pod \"2838ebe2-712b-4d88-881d-c4c17c6308f6\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") "
Oct 30 13:18:16.306874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.306716    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-home\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-home\") pod \"2838ebe2-712b-4d88-881d-c4c17c6308f6\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") "
Oct 30 13:18:16.306874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.306778    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-dshm\") pod \"2838ebe2-712b-4d88-881d-c4c17c6308f6\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") "
Oct 30 13:18:16.306874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.306849    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2838ebe2-712b-4d88-881d-c4c17c6308f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2838ebe2-712b-4d88-881d-c4c17c6308f6-tls-certs\") pod \"2838ebe2-712b-4d88-881d-c4c17c6308f6\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") "
Oct 30 13:18:16.307067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.306884    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-model-cache" pod "2838ebe2-712b-4d88-881d-c4c17c6308f6" (UID: "2838ebe2-712b-4d88-881d-c4c17c6308f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:16.307067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.306953    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2838ebe2-712b-4d88-881d-c4c17c6308f6-kube-api-access-l489f\" (UniqueName: \"kubernetes.io/projected/2838ebe2-712b-4d88-881d-c4c17c6308f6-kube-api-access-l489f\") pod \"2838ebe2-712b-4d88-881d-c4c17c6308f6\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") "
Oct 30 13:18:16.307067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.307006    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-tmp-dir\") pod \"2838ebe2-712b-4d88-881d-c4c17c6308f6\" (UID: \"2838ebe2-712b-4d88-881d-c4c17c6308f6\") "
Oct 30 13:18:16.307272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.307067    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-home" pod "2838ebe2-712b-4d88-881d-c4c17c6308f6" (UID: "2838ebe2-712b-4d88-881d-c4c17c6308f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:16.308555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.307997    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:16.308555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.308020    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:16.308555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.308272    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-tmp-dir" pod "2838ebe2-712b-4d88-881d-c4c17c6308f6" (UID: "2838ebe2-712b-4d88-881d-c4c17c6308f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:16.310760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.310729    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2838ebe2-712b-4d88-881d-c4c17c6308f6-kube-api-access-l489f" pod "2838ebe2-712b-4d88-881d-c4c17c6308f6" (UID: "2838ebe2-712b-4d88-881d-c4c17c6308f6"). InnerVolumeSpecName "kube-api-access-l489f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:16.316514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.316488    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2838ebe2-712b-4d88-881d-c4c17c6308f6-tls-certs" pod "2838ebe2-712b-4d88-881d-c4c17c6308f6" (UID: "2838ebe2-712b-4d88-881d-c4c17c6308f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:16.316601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.316565    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-dshm" pod "2838ebe2-712b-4d88-881d-c4c17c6308f6" (UID: "2838ebe2-712b-4d88-881d-c4c17c6308f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:16.409445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.409354    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l489f\" (UniqueName: \"kubernetes.io/projected/2838ebe2-712b-4d88-881d-c4c17c6308f6-kube-api-access-l489f\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:16.409445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.409390    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:16.409445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.409405    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2838ebe2-712b-4d88-881d-c4c17c6308f6-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:16.409445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.409416    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2838ebe2-712b-4d88-881d-c4c17c6308f6-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:16.553959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.553921    4993 generic.go:356] "Generic (PLEG): container finished" podID="2838ebe2-712b-4d88-881d-c4c17c6308f6" containerID="6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4" exitCode=0
Oct 30 13:18:16.554143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.553973    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"
Oct 30 13:18:16.554143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.554015    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs" event={"ID":"2838ebe2-712b-4d88-881d-c4c17c6308f6","Type":"ContainerDied","Data":"6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4"}
Oct 30 13:18:16.554143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.554062    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs" event={"ID":"2838ebe2-712b-4d88-881d-c4c17c6308f6","Type":"ContainerDied","Data":"b4d16667027f87b080ee2c7844078ce04eab4bff3b4f60d1dbf2c80fdc9eb9f1"}
Oct 30 13:18:16.554143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.554084    4993 scope.go:122] "RemoveContainer" containerID="6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4"
Oct 30 13:18:16.574419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.574399    4993 scope.go:122] "RemoveContainer" containerID="6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4"
Oct 30 13:18:16.574769 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:18:16.574745    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4\": container with ID starting with 6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4 not found: ID does not exist" containerID="6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4"
Oct 30 13:18:16.574869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.574780    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4"} err="failed to get container status \"6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4\": rpc error: code = NotFound desc = could not find container \"6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4\": container with ID starting with 6ab5a0d8c7001cabf447f424f32361d36edc79abae097709bb394e10f652b3b4 not found: ID does not exist"
Oct 30 13:18:16.610754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.610715    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"]
Oct 30 13:18:16.624254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:16.624044    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/auth-test-model-92855c-kserve-557cc7f749-txlhs"]
Oct 30 13:18:17.713229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:17.709640    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2838ebe2-712b-4d88-881d-c4c17c6308f6" path="/var/lib/kubelet/pods/2838ebe2-712b-4d88-881d-c4c17c6308f6/volumes"
Oct 30 13:18:18.197156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.197128    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:18:18.259665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.259606    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba11523-2456-4473-bb55-88f02e0bc8dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba11523-2456-4473-bb55-88f02e0bc8dc-istiod-ca-cert\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.259890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.259681    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-token\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-token\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.259890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.259720    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-kube-api-access-655vp\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-kube-api-access-655vp\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.259890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.259768    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-certs\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.259890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.259822    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-credential-socket\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.259890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.259886    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-podinfo\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.260252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.259997    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-socket\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.260252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.260030    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-data\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.260252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.260110    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-envoy\") pod \"1ba11523-2456-4473-bb55-88f02e0bc8dc\" (UID: \"1ba11523-2456-4473-bb55-88f02e0bc8dc\") "
Oct 30 13:18:18.264947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.261320    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-certs" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:18.264947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.261531    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-credential-socket" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:18.264947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.261574    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba11523-2456-4473-bb55-88f02e0bc8dc-istiod-ca-cert" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:18.264947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.261716    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-socket" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:18.264947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.261893    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-data" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:18.264947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.264354    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-envoy" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:18.264947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.264450    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-kube-api-access-655vp" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "kube-api-access-655vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:18.264947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.264626    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-podinfo" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:18:18.267625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.267598    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-token" pod "1ba11523-2456-4473-bb55-88f02e0bc8dc" (UID: "1ba11523-2456-4473-bb55-88f02e0bc8dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:18.362662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362570    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.362662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362612    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.362662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362627    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.362662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362642    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.362662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362655    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.362962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362668    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.362962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362681    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba11523-2456-4473-bb55-88f02e0bc8dc-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.362962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362693    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.362962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.362706    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655vp\" (UniqueName: \"kubernetes.io/projected/1ba11523-2456-4473-bb55-88f02e0bc8dc-kube-api-access-655vp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.591227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.590703    4993 generic.go:356] "Generic (PLEG): container finished" podID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerID="7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd" exitCode=0
Oct 30 13:18:18.591227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.590844    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" event={"ID":"1ba11523-2456-4473-bb55-88f02e0bc8dc","Type":"ContainerDied","Data":"7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd"}
Oct 30 13:18:18.591227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.590936    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5" event={"ID":"1ba11523-2456-4473-bb55-88f02e0bc8dc","Type":"ContainerDied","Data":"bbccf0ec23e9521015f7eb55a37eb50bcc00234fd89ac4030d89e8006b5e1707"}
Oct 30 13:18:18.591227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.590977    4993 scope.go:122] "RemoveContainer" containerID="7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd"
Oct 30 13:18:18.591227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.591167    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"
Oct 30 13:18:18.605459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.605436    4993 scope.go:122] "RemoveContainer" containerID="7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd"
Oct 30 13:18:18.605785 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:18:18.605764    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd\": container with ID starting with 7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd not found: ID does not exist" containerID="7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd"
Oct 30 13:18:18.605832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.605794    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd"} err="failed to get container status \"7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd\": rpc error: code = NotFound desc = could not find container \"7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd\": container with ID starting with 7f520e7b89a3a2a76ff32d1d3bc23506290c52a425cd76a361dd032801cc52cd not found: ID does not exist"
Oct 30 13:18:18.645813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.645735    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"]
Oct 30 13:18:18.658333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:18.658296    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-xzjs5"]
Oct 30 13:18:19.416367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:19.416330    4993 scope.go:122] "RemoveContainer" containerID="3a6e6850f70226d50bb64bfd5418a988bce8626be7c6c1639db65349e35bbf8c"
Oct 30 13:18:19.427797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:19.427772    4993 scope.go:122] "RemoveContainer" containerID="5c0f99651478b100c1d35ed5fea7d74a158de29b7d4f331db5dc028bd7107245"
Oct 30 13:18:19.437680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:19.437656    4993 scope.go:122] "RemoveContainer" containerID="a3c0c1c62f87426677837e3508d0c4df70f007c5729c787b2545af5fefdb370a"
Oct 30 13:18:19.447143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:19.447122    4993 scope.go:122] "RemoveContainer" containerID="82591f1c61cf5b9ccced2d378ccfd260fb718b21a32a416e8911278d55775b29"
Oct 30 13:18:19.456439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:19.456417    4993 scope.go:122] "RemoveContainer" containerID="cacf17609a0c8c5f45781ebfa550a0e3d4bdfa82b352550f4232b39bccd60558"
Oct 30 13:18:19.465370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:19.465348    4993 scope.go:122] "RemoveContainer" containerID="8d0d510d66f68838d560b1e7ca9113b6630babdbe6d5c3e8da5871ddc1e70db5"
Oct 30 13:18:19.691648 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:19.691570    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" path="/var/lib/kubelet/pods/1ba11523-2456-4473-bb55-88f02e0bc8dc/volumes"
Oct 30 13:18:20.964457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.964418    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"]
Oct 30 13:18:20.964960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.964944    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4578924e-2c82-4845-a56c-149ac69bfd21" containerName="main"
Oct 30 13:18:20.964960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.964959    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4578924e-2c82-4845-a56c-149ac69bfd21" containerName="main"
Oct 30 13:18:20.965076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.964974    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy"
Oct 30 13:18:20.965076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.964979    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy"
Oct 30 13:18:20.965076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.964997    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c661b0b0-3112-4103-8726-a9c393b027e2" containerName="revoke-keys"
Oct 30 13:18:20.965076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.965002    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c661b0b0-3112-4103-8726-a9c393b027e2" containerName="revoke-keys"
Oct 30 13:18:20.965076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.965015    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2838ebe2-712b-4d88-881d-c4c17c6308f6" containerName="main"
Oct 30 13:18:20.965076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.965020    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2838ebe2-712b-4d88-881d-c4c17c6308f6" containerName="main"
Oct 30 13:18:20.965512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.965098    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c661b0b0-3112-4103-8726-a9c393b027e2" containerName="revoke-keys"
Oct 30 13:18:20.965512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.965107    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2838ebe2-712b-4d88-881d-c4c17c6308f6" containerName="main"
Oct 30 13:18:20.965512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.965118    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4578924e-2c82-4845-a56c-149ac69bfd21" containerName="main"
Oct 30 13:18:20.965512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.965125    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba11523-2456-4473-bb55-88f02e0bc8dc" containerName="istio-proxy"
Oct 30 13:18:20.970569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.970546    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.001257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.997267    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:18:21.001257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.997321    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:18:21.001257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:20.997321    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:18:21.003886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.003852    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"]
Oct 30 13:18:21.004058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.004039    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:18:21.101493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.101460    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.101762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.101732    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.101913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.101784    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8gq\" (UniqueName: \"kubernetes.io/projected/3d70f700-0873-4ce2-a36c-272859f50ab8-kube-api-access-lw8gq\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.202746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.202710    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.202974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.202954    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.203034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.203009    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8gq\" (UniqueName: \"kubernetes.io/projected/3d70f700-0873-4ce2-a36c-272859f50ab8-kube-api-access-lw8gq\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.203520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.203496    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.205393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.205365    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.226487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.226422    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8gq\" (UniqueName: \"kubernetes.io/projected/3d70f700-0873-4ce2-a36c-272859f50ab8-kube-api-access-lw8gq\") pod \"maas-api-revoke-sub-0b05e99a-6vj24\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.283978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.283931    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:21.594292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.594159    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"]
Oct 30 13:18:21.597626 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:18:21.597581    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d70f700_0873_4ce2_a36c_272859f50ab8.slice/crio-ba65556a54636f4aaa080ae3dff937b9ecc7124ebce19d4d32fd127b72964ad6 WatchSource:0}: Error finding container ba65556a54636f4aaa080ae3dff937b9ecc7124ebce19d4d32fd127b72964ad6: Status 404 returned error can't find the container with id ba65556a54636f4aaa080ae3dff937b9ecc7124ebce19d4d32fd127b72964ad6
Oct 30 13:18:21.612260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:21.612225    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24" event={"ID":"3d70f700-0873-4ce2-a36c-272859f50ab8","Type":"ContainerStarted","Data":"ba65556a54636f4aaa080ae3dff937b9ecc7124ebce19d4d32fd127b72964ad6"}
Oct 30 13:18:22.620908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:22.620871    4993 generic.go:356] "Generic (PLEG): container finished" podID="3d70f700-0873-4ce2-a36c-272859f50ab8" containerID="d086bf9effa59850366f7ff5931dab3148dc3c416ac06152cb826d8b6c457672" exitCode=0
Oct 30 13:18:22.621375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:22.620939    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24" event={"ID":"3d70f700-0873-4ce2-a36c-272859f50ab8","Type":"ContainerDied","Data":"d086bf9effa59850366f7ff5931dab3148dc3c416ac06152cb826d8b6c457672"}
Oct 30 13:18:23.709096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.709062    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"]
Oct 30 13:18:23.714270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.713693    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.762378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.762344    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"]
Oct 30 13:18:23.800751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.800722    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:23.844714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.844676    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.844912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.844769    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5v7m\" (UniqueName: \"kubernetes.io/projected/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-kube-api-access-b5v7m\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.844912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.844896    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.946089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.946005    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-xks-ca\") pod \"3d70f700-0873-4ce2-a36c-272859f50ab8\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") "
Oct 30 13:18:23.946291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.946099    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d70f700-0873-4ce2-a36c-272859f50ab8-kube-api-access-lw8gq\" (UniqueName: \"kubernetes.io/projected/3d70f700-0873-4ce2-a36c-272859f50ab8-kube-api-access-lw8gq\") pod \"3d70f700-0873-4ce2-a36c-272859f50ab8\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") "
Oct 30 13:18:23.946291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.946150    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-openshift-ca\") pod \"3d70f700-0873-4ce2-a36c-272859f50ab8\" (UID: \"3d70f700-0873-4ce2-a36c-272859f50ab8\") "
Oct 30 13:18:23.946618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.946587    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-openshift-ca" pod "3d70f700-0873-4ce2-a36c-272859f50ab8" (UID: "3d70f700-0873-4ce2-a36c-272859f50ab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:23.946618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.946604    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.946797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.946672    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5v7m\" (UniqueName: \"kubernetes.io/projected/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-kube-api-access-b5v7m\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.946797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.946775    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.946959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.946943    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:23.947449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.947343    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.948659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.948638    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-xks-ca" pod "3d70f700-0873-4ce2-a36c-272859f50ab8" (UID: "3d70f700-0873-4ce2-a36c-272859f50ab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:23.949421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.949385    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d70f700-0873-4ce2-a36c-272859f50ab8-kube-api-access-lw8gq" pod "3d70f700-0873-4ce2-a36c-272859f50ab8" (UID: "3d70f700-0873-4ce2-a36c-272859f50ab8"). InnerVolumeSpecName "kube-api-access-lw8gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:23.949539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.949507    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:23.971375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:23.971350    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5v7m\" (UniqueName: \"kubernetes.io/projected/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-kube-api-access-b5v7m\") pod \"maas-api-revoke-sub-099b5205-l8r8d\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:24.045516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:24.045480    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:24.048356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:24.048322    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d70f700-0873-4ce2-a36c-272859f50ab8-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:24.048356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:24.048359    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8gq\" (UniqueName: \"kubernetes.io/projected/3d70f700-0873-4ce2-a36c-272859f50ab8-kube-api-access-lw8gq\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:24.360956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:24.360923    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"]
Oct 30 13:18:24.363653 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:18:24.363619    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2138d10_0883_4a24_b6d5_d1bc847fc8dc.slice/crio-c7907e92f5418fd24290da5963d164b6127daad70f7dde590713eaf886f46a92 WatchSource:0}: Error finding container c7907e92f5418fd24290da5963d164b6127daad70f7dde590713eaf886f46a92: Status 404 returned error can't find the container with id c7907e92f5418fd24290da5963d164b6127daad70f7dde590713eaf886f46a92
Oct 30 13:18:24.639886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:24.639806    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"
Oct 30 13:18:24.639886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:24.639830    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24" event={"ID":"3d70f700-0873-4ce2-a36c-272859f50ab8","Type":"ContainerDied","Data":"ba65556a54636f4aaa080ae3dff937b9ecc7124ebce19d4d32fd127b72964ad6"}
Oct 30 13:18:24.639886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:24.639882    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba65556a54636f4aaa080ae3dff937b9ecc7124ebce19d4d32fd127b72964ad6"
Oct 30 13:18:24.641528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:24.641500    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d" event={"ID":"c2138d10-0883-4a24-b6d5-d1bc847fc8dc","Type":"ContainerStarted","Data":"c7907e92f5418fd24290da5963d164b6127daad70f7dde590713eaf886f46a92"}
Oct 30 13:18:25.652759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:25.652666    4993 generic.go:356] "Generic (PLEG): container finished" podID="c2138d10-0883-4a24-b6d5-d1bc847fc8dc" containerID="358ab6e6c4e07bb3393368a1cf2346d5def2b8e6b2eb21e23846ac5427f54530" exitCode=0
Oct 30 13:18:25.652759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:25.652719    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d" event={"ID":"c2138d10-0883-4a24-b6d5-d1bc847fc8dc","Type":"ContainerDied","Data":"358ab6e6c4e07bb3393368a1cf2346d5def2b8e6b2eb21e23846ac5427f54530"}
Oct 30 13:18:26.854883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.854854    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:26.893631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.893590    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-xks-ca\") pod \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") "
Oct 30 13:18:26.893786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.893751    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-kube-api-access-b5v7m\" (UniqueName: \"kubernetes.io/projected/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-kube-api-access-b5v7m\") pod \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") "
Oct 30 13:18:26.893886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.893870    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-openshift-ca\") pod \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\" (UID: \"c2138d10-0883-4a24-b6d5-d1bc847fc8dc\") "
Oct 30 13:18:26.895339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.895304    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-openshift-ca" pod "c2138d10-0883-4a24-b6d5-d1bc847fc8dc" (UID: "c2138d10-0883-4a24-b6d5-d1bc847fc8dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:26.907265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.897269    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-xks-ca" pod "c2138d10-0883-4a24-b6d5-d1bc847fc8dc" (UID: "c2138d10-0883-4a24-b6d5-d1bc847fc8dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:26.907265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.897671    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-kube-api-access-b5v7m" pod "c2138d10-0883-4a24-b6d5-d1bc847fc8dc" (UID: "c2138d10-0883-4a24-b6d5-d1bc847fc8dc"). InnerVolumeSpecName "kube-api-access-b5v7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:26.996019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.995952    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5v7m\" (UniqueName: \"kubernetes.io/projected/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-kube-api-access-b5v7m\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.996019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.995996    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.996019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:26.996012    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2138d10-0883-4a24-b6d5-d1bc847fc8dc-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:27.673716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:27.673680    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"
Oct 30 13:18:27.673716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:27.673695    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d" event={"ID":"c2138d10-0883-4a24-b6d5-d1bc847fc8dc","Type":"ContainerDied","Data":"c7907e92f5418fd24290da5963d164b6127daad70f7dde590713eaf886f46a92"}
Oct 30 13:18:27.673992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:27.673768    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7907e92f5418fd24290da5963d164b6127daad70f7dde590713eaf886f46a92"
Oct 30 13:18:31.317288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.317255    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"]
Oct 30 13:18:31.317866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.317851    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c2138d10-0883-4a24-b6d5-d1bc847fc8dc" containerName="revoke-keys"
Oct 30 13:18:31.317913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.317867    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2138d10-0883-4a24-b6d5-d1bc847fc8dc" containerName="revoke-keys"
Oct 30 13:18:31.317913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.317880    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3d70f700-0873-4ce2-a36c-272859f50ab8" containerName="revoke-keys"
Oct 30 13:18:31.317913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.317886    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d70f700-0873-4ce2-a36c-272859f50ab8" containerName="revoke-keys"
Oct 30 13:18:31.318014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.317987    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d70f700-0873-4ce2-a36c-272859f50ab8" containerName="revoke-keys"
Oct 30 13:18:31.318014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.318006    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2138d10-0883-4a24-b6d5-d1bc847fc8dc" containerName="revoke-keys"
Oct 30 13:18:31.321532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.321512    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.337532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.337505    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:18:31.349020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.348983    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:18:31.349293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.349278    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:18:31.349421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.349339    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:18:31.355303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.355271    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"]
Oct 30 13:18:31.372600 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.372562    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.372881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.372631    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.372973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.372876    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn8nd\" (UniqueName: \"kubernetes.io/projected/f2b6e425-c818-4bad-aee1-84118551bfeb-kube-api-access-wn8nd\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.474551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.474518    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn8nd\" (UniqueName: \"kubernetes.io/projected/f2b6e425-c818-4bad-aee1-84118551bfeb-kube-api-access-wn8nd\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.474751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.474664    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.474751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.474705    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.475437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.475407    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.477290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.477257    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.500283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.500243    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn8nd\" (UniqueName: \"kubernetes.io/projected/f2b6e425-c818-4bad-aee1-84118551bfeb-kube-api-access-wn8nd\") pod \"maas-api-revoke-sub-593b3a2a-fjpr8\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.636142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.636059    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:31.948967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:31.948786    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"]
Oct 30 13:18:31.953530 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:18:31.953492    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b6e425_c818_4bad_aee1_84118551bfeb.slice/crio-7810b45c8a3c2b1598a12c489dc1be0ac35e8b1cd452fe8a4d526bbad8920fa6 WatchSource:0}: Error finding container 7810b45c8a3c2b1598a12c489dc1be0ac35e8b1cd452fe8a4d526bbad8920fa6: Status 404 returned error can't find the container with id 7810b45c8a3c2b1598a12c489dc1be0ac35e8b1cd452fe8a4d526bbad8920fa6
Oct 30 13:18:32.726294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:32.726175    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8" event={"ID":"f2b6e425-c818-4bad-aee1-84118551bfeb","Type":"ContainerStarted","Data":"ba7bc8055d105ffebeacbca7a4dbc7afa0b388806782da5a38fe7c9a43053d03"}
Oct 30 13:18:32.726294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:32.726243    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8" event={"ID":"f2b6e425-c818-4bad-aee1-84118551bfeb","Type":"ContainerStarted","Data":"7810b45c8a3c2b1598a12c489dc1be0ac35e8b1cd452fe8a4d526bbad8920fa6"}
Oct 30 13:18:32.771826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:32.771758    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8" podStartSLOduration=1.771741428 podStartE2EDuration="1.771741428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:32.766025772 +0000 UTC m=+2777.241745383" watchObservedRunningTime="2026-10-30 13:18:32.771741428 +0000 UTC m=+2777.247461039"
Oct 30 13:18:33.735905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:33.735870    4993 generic.go:356] "Generic (PLEG): container finished" podID="f2b6e425-c818-4bad-aee1-84118551bfeb" containerID="ba7bc8055d105ffebeacbca7a4dbc7afa0b388806782da5a38fe7c9a43053d03" exitCode=0
Oct 30 13:18:33.736347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:33.735940    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8" event={"ID":"f2b6e425-c818-4bad-aee1-84118551bfeb","Type":"ContainerDied","Data":"ba7bc8055d105ffebeacbca7a4dbc7afa0b388806782da5a38fe7c9a43053d03"}
Oct 30 13:18:34.126125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.126049    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"]
Oct 30 13:18:34.130095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.130072    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.150543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.150510    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"]
Oct 30 13:18:34.230626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.230592    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.230847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.230743    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.230955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.230939    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kb84\" (UniqueName: \"kubernetes.io/projected/6adc327e-c516-47ba-ad9f-6bd584d28610-kube-api-access-4kb84\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.332130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.332087    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kb84\" (UniqueName: \"kubernetes.io/projected/6adc327e-c516-47ba-ad9f-6bd584d28610-kube-api-access-4kb84\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.332374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.332219    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.332374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.332308    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.333149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.333121    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.334551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.334523    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.358531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.358498    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kb84\" (UniqueName: \"kubernetes.io/projected/6adc327e-c516-47ba-ad9f-6bd584d28610-kube-api-access-4kb84\") pod \"maas-api-revoke-sub-7ae602ca-xr7hg\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.454284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.453438    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:34.911680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.911645    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"]
Oct 30 13:18:34.913873 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:18:34.913841    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adc327e_c516_47ba_ad9f_6bd584d28610.slice/crio-dc48e9d7ead21907f8bd8e26a96887947ef8d586400ab0dbf2c2337c1c9103c0 WatchSource:0}: Error finding container dc48e9d7ead21907f8bd8e26a96887947ef8d586400ab0dbf2c2337c1c9103c0: Status 404 returned error can't find the container with id dc48e9d7ead21907f8bd8e26a96887947ef8d586400ab0dbf2c2337c1c9103c0
Oct 30 13:18:34.941492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:34.941468    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:35.049930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.049846    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-xks-ca\") pod \"f2b6e425-c818-4bad-aee1-84118551bfeb\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") "
Oct 30 13:18:35.050088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.049960    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-openshift-ca\") pod \"f2b6e425-c818-4bad-aee1-84118551bfeb\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") "
Oct 30 13:18:35.050140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.050099    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b6e425-c818-4bad-aee1-84118551bfeb-kube-api-access-wn8nd\" (UniqueName: \"kubernetes.io/projected/f2b6e425-c818-4bad-aee1-84118551bfeb-kube-api-access-wn8nd\") pod \"f2b6e425-c818-4bad-aee1-84118551bfeb\" (UID: \"f2b6e425-c818-4bad-aee1-84118551bfeb\") "
Oct 30 13:18:35.050387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.050361    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-openshift-ca" pod "f2b6e425-c818-4bad-aee1-84118551bfeb" (UID: "f2b6e425-c818-4bad-aee1-84118551bfeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:35.050654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.050639    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:35.052070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.052048    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-xks-ca" pod "f2b6e425-c818-4bad-aee1-84118551bfeb" (UID: "f2b6e425-c818-4bad-aee1-84118551bfeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:35.052226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.052186    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b6e425-c818-4bad-aee1-84118551bfeb-kube-api-access-wn8nd" pod "f2b6e425-c818-4bad-aee1-84118551bfeb" (UID: "f2b6e425-c818-4bad-aee1-84118551bfeb"). InnerVolumeSpecName "kube-api-access-wn8nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:35.152168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.152135    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn8nd\" (UniqueName: \"kubernetes.io/projected/f2b6e425-c818-4bad-aee1-84118551bfeb-kube-api-access-wn8nd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:35.152168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.152164    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b6e425-c818-4bad-aee1-84118551bfeb-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:35.797752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.797693    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8" event={"ID":"f2b6e425-c818-4bad-aee1-84118551bfeb","Type":"ContainerDied","Data":"7810b45c8a3c2b1598a12c489dc1be0ac35e8b1cd452fe8a4d526bbad8920fa6"}
Oct 30 13:18:35.798052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.798021    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7810b45c8a3c2b1598a12c489dc1be0ac35e8b1cd452fe8a4d526bbad8920fa6"
Oct 30 13:18:35.798311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.798264    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"
Oct 30 13:18:35.803041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.801790    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg" event={"ID":"6adc327e-c516-47ba-ad9f-6bd584d28610","Type":"ContainerStarted","Data":"984659af2954b214b7be0cc1ea453f1cf5c7082d6318c0af023778e56a1a720f"}
Oct 30 13:18:35.803041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.801839    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg" event={"ID":"6adc327e-c516-47ba-ad9f-6bd584d28610","Type":"ContainerStarted","Data":"dc48e9d7ead21907f8bd8e26a96887947ef8d586400ab0dbf2c2337c1c9103c0"}
Oct 30 13:18:35.910245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:35.893593    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg" podStartSLOduration=1.8935703400000001 podStartE2EDuration="1.89357034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:35.868754495 +0000 UTC m=+2780.344474105" watchObservedRunningTime="2026-10-30 13:18:35.89357034 +0000 UTC m=+2780.369289952"
Oct 30 13:18:36.814420 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:36.814375    4993 generic.go:356] "Generic (PLEG): container finished" podID="6adc327e-c516-47ba-ad9f-6bd584d28610" containerID="984659af2954b214b7be0cc1ea453f1cf5c7082d6318c0af023778e56a1a720f" exitCode=0
Oct 30 13:18:36.814877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:36.814475    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg" event={"ID":"6adc327e-c516-47ba-ad9f-6bd584d28610","Type":"ContainerDied","Data":"984659af2954b214b7be0cc1ea453f1cf5c7082d6318c0af023778e56a1a720f"}
Oct 30 13:18:38.018086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.018052    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:38.147346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.147248    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-xks-ca\") pod \"6adc327e-c516-47ba-ad9f-6bd584d28610\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") "
Oct 30 13:18:38.147534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.147462    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-openshift-ca\") pod \"6adc327e-c516-47ba-ad9f-6bd584d28610\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") "
Oct 30 13:18:38.147534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.147529    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adc327e-c516-47ba-ad9f-6bd584d28610-kube-api-access-4kb84\" (UniqueName: \"kubernetes.io/projected/6adc327e-c516-47ba-ad9f-6bd584d28610-kube-api-access-4kb84\") pod \"6adc327e-c516-47ba-ad9f-6bd584d28610\" (UID: \"6adc327e-c516-47ba-ad9f-6bd584d28610\") "
Oct 30 13:18:38.147893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.147866    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-openshift-ca" pod "6adc327e-c516-47ba-ad9f-6bd584d28610" (UID: "6adc327e-c516-47ba-ad9f-6bd584d28610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:38.149632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.149582    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-xks-ca" pod "6adc327e-c516-47ba-ad9f-6bd584d28610" (UID: "6adc327e-c516-47ba-ad9f-6bd584d28610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:38.149773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.149747    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adc327e-c516-47ba-ad9f-6bd584d28610-kube-api-access-4kb84" pod "6adc327e-c516-47ba-ad9f-6bd584d28610" (UID: "6adc327e-c516-47ba-ad9f-6bd584d28610"). InnerVolumeSpecName "kube-api-access-4kb84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:38.249112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.249072    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kb84\" (UniqueName: \"kubernetes.io/projected/6adc327e-c516-47ba-ad9f-6bd584d28610-kube-api-access-4kb84\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:38.249112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.249109    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:38.249112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.249120    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adc327e-c516-47ba-ad9f-6bd584d28610-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:38.575748 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:18:38.575702    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:18:38.836153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.836071    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"
Oct 30 13:18:38.836153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.836091    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg" event={"ID":"6adc327e-c516-47ba-ad9f-6bd584d28610","Type":"ContainerDied","Data":"dc48e9d7ead21907f8bd8e26a96887947ef8d586400ab0dbf2c2337c1c9103c0"}
Oct 30 13:18:38.836153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:38.836147    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc48e9d7ead21907f8bd8e26a96887947ef8d586400ab0dbf2c2337c1c9103c0"
Oct 30 13:18:43.950773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:43.950730    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"]
Oct 30 13:18:54.551534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:54.551484    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"]
Oct 30 13:18:55.280955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:55.280913    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"]
Oct 30 13:18:55.281261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:55.281232    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy" containerID="cri-o://3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566" gracePeriod=30
Oct 30 13:18:58.996873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:58.996827    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf" podUID="97b9f157-a886-415e-b6f1-dfa1364462d3" containerName="main" containerID="cri-o://165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b" gracePeriod=45
Oct 30 13:18:59.385000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.384973    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:18:59.518668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.518570    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-dshm\") pod \"97b9f157-a886-415e-b6f1-dfa1364462d3\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") "
Oct 30 13:18:59.518829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.518786    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b9f157-a886-415e-b6f1-dfa1364462d3-kube-api-access-mztdb\" (UniqueName: \"kubernetes.io/projected/97b9f157-a886-415e-b6f1-dfa1364462d3-kube-api-access-mztdb\") pod \"97b9f157-a886-415e-b6f1-dfa1364462d3\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") "
Oct 30 13:18:59.518883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.518826    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-model-cache\") pod \"97b9f157-a886-415e-b6f1-dfa1364462d3\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") "
Oct 30 13:18:59.518963 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.518935    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-tmp-dir\") pod \"97b9f157-a886-415e-b6f1-dfa1364462d3\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") "
Oct 30 13:18:59.519027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.518986    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b9f157-a886-415e-b6f1-dfa1364462d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/97b9f157-a886-415e-b6f1-dfa1364462d3-tls-certs\") pod \"97b9f157-a886-415e-b6f1-dfa1364462d3\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") "
Oct 30 13:18:59.519092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.519053    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-home\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-home\") pod \"97b9f157-a886-415e-b6f1-dfa1364462d3\" (UID: \"97b9f157-a886-415e-b6f1-dfa1364462d3\") "
Oct 30 13:18:59.519527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.519218    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-model-cache" pod "97b9f157-a886-415e-b6f1-dfa1364462d3" (UID: "97b9f157-a886-415e-b6f1-dfa1364462d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:59.519527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.519355    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-tmp-dir" pod "97b9f157-a886-415e-b6f1-dfa1364462d3" (UID: "97b9f157-a886-415e-b6f1-dfa1364462d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:59.519527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.519423    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-home" pod "97b9f157-a886-415e-b6f1-dfa1364462d3" (UID: "97b9f157-a886-415e-b6f1-dfa1364462d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:59.519754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.519659    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:59.519754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.519681    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:59.519754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.519697    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:59.521426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.521398    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b9f157-a886-415e-b6f1-dfa1364462d3-tls-certs" pod "97b9f157-a886-415e-b6f1-dfa1364462d3" (UID: "97b9f157-a886-415e-b6f1-dfa1364462d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:59.521572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.521551    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-dshm" pod "97b9f157-a886-415e-b6f1-dfa1364462d3" (UID: "97b9f157-a886-415e-b6f1-dfa1364462d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:59.521634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.521578    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b9f157-a886-415e-b6f1-dfa1364462d3-kube-api-access-mztdb" pod "97b9f157-a886-415e-b6f1-dfa1364462d3" (UID: "97b9f157-a886-415e-b6f1-dfa1364462d3"). InnerVolumeSpecName "kube-api-access-mztdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:59.621054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.621004    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mztdb\" (UniqueName: \"kubernetes.io/projected/97b9f157-a886-415e-b6f1-dfa1364462d3-kube-api-access-mztdb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:59.621054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.621046    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b9f157-a886-415e-b6f1-dfa1364462d3-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:59.621054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.621057    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b9f157-a886-415e-b6f1-dfa1364462d3-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:18:59.746703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.746667    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"]
Oct 30 13:18:59.746932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:18:59.746909    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy" containerID="cri-o://b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267" gracePeriod=30
Oct 30 13:19:00.059358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.059323    4993 generic.go:356] "Generic (PLEG): container finished" podID="97b9f157-a886-415e-b6f1-dfa1364462d3" containerID="165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b" exitCode=0
Oct 30 13:19:00.059872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.059370    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"
Oct 30 13:19:00.059872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.059406    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf" event={"ID":"97b9f157-a886-415e-b6f1-dfa1364462d3","Type":"ContainerDied","Data":"165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b"}
Oct 30 13:19:00.059872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.059460    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf" event={"ID":"97b9f157-a886-415e-b6f1-dfa1364462d3","Type":"ContainerDied","Data":"44d04176d349565e5b7aef8a8cb001a69d476135bc9fdb623020a8931fe73840"}
Oct 30 13:19:00.059872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.059483    4993 scope.go:122] "RemoveContainer" containerID="165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b"
Oct 30 13:19:00.071079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.071053    4993 scope.go:122] "RemoveContainer" containerID="165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b"
Oct 30 13:19:00.071559 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:19:00.071528    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b\": container with ID starting with 165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b not found: ID does not exist" containerID="165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b"
Oct 30 13:19:00.071696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.071570    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b"} err="failed to get container status \"165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b\": rpc error: code = NotFound desc = could not find container \"165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b\": container with ID starting with 165b5ad6392c974b9da81355579ee5488d9a735b870b34174512fbfefd7b2d8b not found: ID does not exist"
Oct 30 13:19:00.114016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.113977    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"]
Oct 30 13:19:00.118728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.118700    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-bd2aaf/auth-test-model-bd2aaf-kserve-f4ddf85fc-4rfcf"]
Oct 30 13:19:00.594003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.593973    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:19:00.736593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736558    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-kube-api-access-f8crz\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-kube-api-access-f8crz\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.736778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736626    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-credential-socket\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.736778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736684    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-data\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.736778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736746    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-certs\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.736978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736803    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-socket\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.736978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736880    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-podinfo\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.736978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736910    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aae0a50-1547-441e-9ab7-3231b86ccd73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aae0a50-1547-441e-9ab7-3231b86ccd73-istiod-ca-cert\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.736978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736935    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-credential-socket" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.736978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.736951    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-data" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.737307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.737019    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-envoy\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.737307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.737123    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-token\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-token\") pod \"6aae0a50-1547-441e-9ab7-3231b86ccd73\" (UID: \"6aae0a50-1547-441e-9ab7-3231b86ccd73\") "
Oct 30 13:19:00.737307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.737178    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-certs" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.737307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.737108    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-socket" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.737520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.737309    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aae0a50-1547-441e-9ab7-3231b86ccd73-istiod-ca-cert" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:00.737989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.737959    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.737989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.737988    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.738220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.738003    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.738220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.738048    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.738220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.738062    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aae0a50-1547-441e-9ab7-3231b86ccd73-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.739420 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.739385    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-podinfo" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:19:00.739587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.739565    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-envoy" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.739786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.739759    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-kube-api-access-f8crz" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "kube-api-access-f8crz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:00.739906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.739799    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-token" pod "6aae0a50-1547-441e-9ab7-3231b86ccd73" (UID: "6aae0a50-1547-441e-9ab7-3231b86ccd73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:00.839717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.839617    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.839717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.839650    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.839717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.839661    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.839717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:00.839672    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8crz\" (UniqueName: \"kubernetes.io/projected/6aae0a50-1547-441e-9ab7-3231b86ccd73-kube-api-access-f8crz\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:01.071778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.071737    4993 generic.go:356] "Generic (PLEG): container finished" podID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerID="3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566" exitCode=0
Oct 30 13:19:01.072310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.071792    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"
Oct 30 13:19:01.072310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.071823    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" event={"ID":"6aae0a50-1547-441e-9ab7-3231b86ccd73","Type":"ContainerDied","Data":"3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566"}
Oct 30 13:19:01.072310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.071884    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h" event={"ID":"6aae0a50-1547-441e-9ab7-3231b86ccd73","Type":"ContainerDied","Data":"d13c1a8aabeea7eb97461c060d4e4c35fe4dc81bcb8e36b6a1b53a732fbf3745"}
Oct 30 13:19:01.072310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.071906    4993 scope.go:122] "RemoveContainer" containerID="3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566"
Oct 30 13:19:01.085162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.085135    4993 scope.go:122] "RemoveContainer" containerID="3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566"
Oct 30 13:19:01.085565 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:19:01.085526    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566\": container with ID starting with 3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566 not found: ID does not exist" containerID="3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566"
Oct 30 13:19:01.085686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.085567    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566"} err="failed to get container status \"3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566\": rpc error: code = NotFound desc = could not find container \"3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566\": container with ID starting with 3b934d4636e1e2ad40dfda6961005794c45106c37c05dc5244eaf9356391c566 not found: ID does not exist"
Oct 30 13:19:01.127906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.127805    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"]
Oct 30 13:19:01.139496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.139456    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-6cq8h"]
Oct 30 13:19:01.691806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.691773    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" path="/var/lib/kubelet/pods/6aae0a50-1547-441e-9ab7-3231b86ccd73/volumes"
Oct 30 13:19:01.692292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:01.692276    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b9f157-a886-415e-b6f1-dfa1364462d3" path="/var/lib/kubelet/pods/97b9f157-a886-415e-b6f1-dfa1364462d3/volumes"
Oct 30 13:19:05.030595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.030536    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 13:19:05.121881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.121835    4993 generic.go:356] "Generic (PLEG): container finished" podID="97edabe4-74b8-4125-a2be-642e0470eba5" containerID="b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267" exitCode=0
Oct 30 13:19:05.122055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.121889    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"
Oct 30 13:19:05.122055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.121889    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" event={"ID":"97edabe4-74b8-4125-a2be-642e0470eba5","Type":"ContainerDied","Data":"b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267"}
Oct 30 13:19:05.122055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.122018    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b" event={"ID":"97edabe4-74b8-4125-a2be-642e0470eba5","Type":"ContainerDied","Data":"9a16e9de38e65d03680f5eb7816e32588d07e6963be831e28ce441a2c7477a76"}
Oct 30 13:19:05.122055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.122040    4993 scope.go:122] "RemoveContainer" containerID="b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267"
Oct 30 13:19:05.123186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123146    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-kube-api-access-4v5rz\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-kube-api-access-4v5rz\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.123303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123189    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-data\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.123303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123253    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97edabe4-74b8-4125-a2be-642e0470eba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97edabe4-74b8-4125-a2be-642e0470eba5-istiod-ca-cert\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.123385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123330    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-socket\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.123430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123384    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97edabe4-74b8-4125-a2be-642e0470eba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97edabe4-74b8-4125-a2be-642e0470eba5-istio-podinfo\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.123475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123461    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-credential-socket\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.123523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123490    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-istio-token\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-istio-token\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.123523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123514    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-envoy\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.123600 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.123537    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-certs\") pod \"97edabe4-74b8-4125-a2be-642e0470eba5\" (UID: \"97edabe4-74b8-4125-a2be-642e0470eba5\") "
Oct 30 13:19:05.124173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.124154    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-certs" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:05.124417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.124391    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-socket" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:05.124698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.124671    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-data" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:05.124794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.124739    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97edabe4-74b8-4125-a2be-642e0470eba5-istiod-ca-cert" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:05.124866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.124839    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-credential-socket" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:05.127291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.127258    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97edabe4-74b8-4125-a2be-642e0470eba5-istio-podinfo" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:19:05.128474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.128439    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-istio-token" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:05.143343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.133716    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-envoy" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:05.143343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.133778    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-kube-api-access-4v5rz" pod "97edabe4-74b8-4125-a2be-642e0470eba5" (UID: "97edabe4-74b8-4125-a2be-642e0470eba5"). InnerVolumeSpecName "kube-api-access-4v5rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:05.162147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.162117    4993 scope.go:122] "RemoveContainer" containerID="b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267"
Oct 30 13:19:05.162564 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:19:05.162537    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267\": container with ID starting with b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267 not found: ID does not exist" containerID="b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267"
Oct 30 13:19:05.162633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.162577    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267"} err="failed to get container status \"b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267\": rpc error: code = NotFound desc = could not find container \"b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267\": container with ID starting with b0fca1ecd500b7f381ec1861dc5fc2b83e600c4bbfa00dc5d75083d32daca267 not found: ID does not exist"
Oct 30 13:19:05.225405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225373    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97edabe4-74b8-4125-a2be-642e0470eba5-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.225405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225401    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.225405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225411    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.225658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225420    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.225658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225430    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.225658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225438    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v5rz\" (UniqueName: \"kubernetes.io/projected/97edabe4-74b8-4125-a2be-642e0470eba5-kube-api-access-4v5rz\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.225658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225447    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.225658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225454    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97edabe4-74b8-4125-a2be-642e0470eba5-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.225658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.225462    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97edabe4-74b8-4125-a2be-642e0470eba5-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:05.474626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.474594    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"]
Oct 30 13:19:05.507224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.504431    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-2v66b"]
Oct 30 13:19:05.694890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:05.694804    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" path="/var/lib/kubelet/pods/97edabe4-74b8-4125-a2be-642e0470eba5/volumes"
Oct 30 13:19:08.567861 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:19:08.567824    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:19:09.582294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:09.582254    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc" podUID="d9d11f0f-84d6-462c-89f5-8c6ec6b06871" containerName="main" containerID="cri-o://d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a" gracePeriod=45
Oct 30 13:19:09.638376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:09.638024    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc" podUID="d9d11f0f-84d6-462c-89f5-8c6ec6b06871" containerName="main" probeResult="failure" output="Get \"https://10.132.0.202:8000/ready\": dial tcp 10.132.0.202:8000: connect: connection refused"
Oct 30 13:19:10.009238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.009184    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:19:10.043892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.043855    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tmp-dir\") pod \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") "
Oct 30 13:19:10.044099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044016    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-model-cache\") pod \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") "
Oct 30 13:19:10.044099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044042    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-home\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-home\") pod \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") "
Oct 30 13:19:10.044349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044102    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tls-certs\") pod \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") "
Oct 30 13:19:10.044349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044160    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-dshm\") pod \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") "
Oct 30 13:19:10.044349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044176    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tmp-dir" pod "d9d11f0f-84d6-462c-89f5-8c6ec6b06871" (UID: "d9d11f0f-84d6-462c-89f5-8c6ec6b06871"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:10.044349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044195    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-kube-api-access-bkvnn\" (UniqueName: \"kubernetes.io/projected/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-kube-api-access-bkvnn\") pod \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\" (UID: \"d9d11f0f-84d6-462c-89f5-8c6ec6b06871\") "
Oct 30 13:19:10.044503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044364    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-model-cache" pod "d9d11f0f-84d6-462c-89f5-8c6ec6b06871" (UID: "d9d11f0f-84d6-462c-89f5-8c6ec6b06871"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:10.044503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044401    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-home" pod "d9d11f0f-84d6-462c-89f5-8c6ec6b06871" (UID: "d9d11f0f-84d6-462c-89f5-8c6ec6b06871"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:10.044755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044730    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:10.044755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044755    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:10.044935 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.044768    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:10.046648 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.046620    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-kube-api-access-bkvnn" pod "d9d11f0f-84d6-462c-89f5-8c6ec6b06871" (UID: "d9d11f0f-84d6-462c-89f5-8c6ec6b06871"). InnerVolumeSpecName "kube-api-access-bkvnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:10.046856 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.046832    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-dshm" pod "d9d11f0f-84d6-462c-89f5-8c6ec6b06871" (UID: "d9d11f0f-84d6-462c-89f5-8c6ec6b06871"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:10.046960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.046860    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tls-certs" pod "d9d11f0f-84d6-462c-89f5-8c6ec6b06871" (UID: "d9d11f0f-84d6-462c-89f5-8c6ec6b06871"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:10.162618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.162316    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:10.162618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.162353    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:10.162618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.162368    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkvnn\" (UniqueName: \"kubernetes.io/projected/d9d11f0f-84d6-462c-89f5-8c6ec6b06871-kube-api-access-bkvnn\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:19:10.239141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.239099    4993 generic.go:356] "Generic (PLEG): container finished" podID="d9d11f0f-84d6-462c-89f5-8c6ec6b06871" containerID="d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a" exitCode=0
Oct 30 13:19:10.239141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.239141    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"
Oct 30 13:19:10.239141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.239231    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc" event={"ID":"d9d11f0f-84d6-462c-89f5-8c6ec6b06871","Type":"ContainerDied","Data":"d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a"}
Oct 30 13:19:10.239684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.239266    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc" event={"ID":"d9d11f0f-84d6-462c-89f5-8c6ec6b06871","Type":"ContainerDied","Data":"7ace8caaf5a370ab85298db2c1eb8be4f51f94170a3b0cad79cfe916abd7184c"}
Oct 30 13:19:10.239684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.239283    4993 scope.go:122] "RemoveContainer" containerID="d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a"
Oct 30 13:19:10.252645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.252583    4993 scope.go:122] "RemoveContainer" containerID="d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a"
Oct 30 13:19:10.253011 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:19:10.252986    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a\": container with ID starting with d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a not found: ID does not exist" containerID="d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a"
Oct 30 13:19:10.253125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.253023    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a"} err="failed to get container status \"d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a\": rpc error: code = NotFound desc = could not find container \"d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a\": container with ID starting with d1b0d78e39cee5839949eb9e1dfee56a062d2d7db82d8e3a77f463aaf0a2db1a not found: ID does not exist"
Oct 30 13:19:10.295894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.295857    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"]
Oct 30 13:19:10.302251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:10.302165    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71fbeb/auth-test-model-71fbeb-kserve-5b77f8997-59pzc"]
Oct 30 13:19:11.441748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.441711    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"]
Oct 30 13:19:11.443050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443002    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d9d11f0f-84d6-462c-89f5-8c6ec6b06871" containerName="main"
Oct 30 13:19:11.443050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443035    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d11f0f-84d6-462c-89f5-8c6ec6b06871" containerName="main"
Oct 30 13:19:11.443050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443055    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f2b6e425-c818-4bad-aee1-84118551bfeb" containerName="revoke-keys"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443064    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b6e425-c818-4bad-aee1-84118551bfeb" containerName="revoke-keys"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443083    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97b9f157-a886-415e-b6f1-dfa1364462d3" containerName="main"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443090    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b9f157-a886-415e-b6f1-dfa1364462d3" containerName="main"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443119    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6adc327e-c516-47ba-ad9f-6bd584d28610" containerName="revoke-keys"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443126    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adc327e-c516-47ba-ad9f-6bd584d28610" containerName="revoke-keys"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443156    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443163    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443178    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443185    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443422    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="97edabe4-74b8-4125-a2be-642e0470eba5" containerName="istio-proxy"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443437    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aae0a50-1547-441e-9ab7-3231b86ccd73" containerName="istio-proxy"
Oct 30 13:19:11.443453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443453    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d11f0f-84d6-462c-89f5-8c6ec6b06871" containerName="main"
Oct 30 13:19:11.444160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443464    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b6e425-c818-4bad-aee1-84118551bfeb" containerName="revoke-keys"
Oct 30 13:19:11.444160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443475    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b9f157-a886-415e-b6f1-dfa1364462d3" containerName="main"
Oct 30 13:19:11.444160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.443489    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adc327e-c516-47ba-ad9f-6bd584d28610" containerName="revoke-keys"
Oct 30 13:19:11.455923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.455884    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.477027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.475298    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"]
Oct 30 13:19:11.480931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.480903    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1957dc-openshift-default-dockercfg-jxcpm\""
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.485858    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-data\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.485959    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-certs\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.485979    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-podinfo\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.486066    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqth\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-kube-api-access-wgqth\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.486140    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-token\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.486165    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istiod-ca-cert\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.486560    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-socket\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.486607    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-credential-socket\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.494060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.486645    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-envoy\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.589254    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqth\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-kube-api-access-wgqth\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.598687    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-token\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.599163    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istiod-ca-cert\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.599262    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-socket\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.599857    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-socket\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.607630    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istiod-ca-cert\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.607795    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-credential-socket\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.607860    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-envoy\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.607921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.607891    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-data\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.613264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.608289    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-credential-socket\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.613264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.608645    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-data\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.613264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.609158    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-certs\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.613264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.610550    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-certs\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.613264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.611107    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-podinfo\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.630608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.628544    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-podinfo\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.630608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.628845    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-envoy\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.647535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.646839    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-token\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.647535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.647475    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqth\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-kube-api-access-wgqth\") pod \"e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") " pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:11.692796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.692693    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d11f0f-84d6-462c-89f5-8c6ec6b06871" path="/var/lib/kubelet/pods/d9d11f0f-84d6-462c-89f5-8c6ec6b06871/volumes"
Oct 30 13:19:11.792992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:11.792949    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:12.155569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:12.155447    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"]
Oct 30 13:19:12.159606 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:19:12.159566    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60c06df_f4cf_43c2_ba80_c100ea8c2b8f.slice/crio-dc1da44ad0586a37b3659c80c1001e3fc1606519bffc73f8a2dbf983fb667ccd WatchSource:0}: Error finding container dc1da44ad0586a37b3659c80c1001e3fc1606519bffc73f8a2dbf983fb667ccd: Status 404 returned error can't find the container with id dc1da44ad0586a37b3659c80c1001e3fc1606519bffc73f8a2dbf983fb667ccd
Oct 30 13:19:12.162577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:12.162536    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:12.162701 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:12.162610    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:12.162701 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:12.162641    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:12.259939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:12.259243    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" event={"ID":"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f","Type":"ContainerStarted","Data":"445105b16cd4eefcd5e228ac2c8d77673656cfd8ad6447f65184bdde3aebcbc6"}
Oct 30 13:19:12.259939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:12.259326    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" event={"ID":"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f","Type":"ContainerStarted","Data":"dc1da44ad0586a37b3659c80c1001e3fc1606519bffc73f8a2dbf983fb667ccd"}
Oct 30 13:19:12.310232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:12.307307    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" podStartSLOduration=1.307286972 podStartE2EDuration="1.307286972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:12.305575233 +0000 UTC m=+2816.781294845" watchObservedRunningTime="2026-10-30 13:19:12.307286972 +0000 UTC m=+2816.783006584"
Oct 30 13:19:12.793244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:12.793164    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:13.799952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:13.799918    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:14.276029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:14.275984    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:14.277071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:14.277047    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:19:29.526614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.526559    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"]
Oct 30 13:19:29.533236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.533171    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.548014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.547958    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-63499e-openshift-default-dockercfg-77r7m\""
Oct 30 13:19:29.566790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.566752    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"]
Oct 30 13:19:29.611745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.611707    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4476629-9b97-427c-82df-d7b54567b2c8-istiod-ca-cert\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.611745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.611753    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-credential-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.612057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.611793    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-certs\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.612057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.611916    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-data\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.612057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.611953    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4476629-9b97-427c-82df-d7b54567b2c8-istio-podinfo\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.612057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.611971    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.612352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.612052    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2842t\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-kube-api-access-2842t\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.612352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.612198    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-envoy\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.612352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.612258    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-istio-token\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731430    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4476629-9b97-427c-82df-d7b54567b2c8-istiod-ca-cert\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731526    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-credential-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731561    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-certs\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731660    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-data\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731699    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4476629-9b97-427c-82df-d7b54567b2c8-istio-podinfo\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731725    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731757    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2842t\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-kube-api-access-2842t\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731862    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-envoy\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.732232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.731885    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-istio-token\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.746232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.732990    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4476629-9b97-427c-82df-d7b54567b2c8-istiod-ca-cert\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.746232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.735285    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.746232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.737376    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-credential-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.746232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.737557    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-certs\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.746232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.737607    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-data\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.821290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.812697    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4476629-9b97-427c-82df-d7b54567b2c8-istio-podinfo\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.827329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.815175    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-envoy\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.827574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.826615    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2842t\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-kube-api-access-2842t\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.848377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.848321    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-istio-token\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:29.885983 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:29.885948    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:30.295363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:30.295324    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"]
Oct 30 13:19:30.297693 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:19:30.297658    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4476629_9b97_427c_82df_d7b54567b2c8.slice/crio-f16e8cb79ce066578e534bab93b9f9b734d06cedaf57eeb06408b9230d735772 WatchSource:0}: Error finding container f16e8cb79ce066578e534bab93b9f9b734d06cedaf57eeb06408b9230d735772: Status 404 returned error can't find the container with id f16e8cb79ce066578e534bab93b9f9b734d06cedaf57eeb06408b9230d735772
Oct 30 13:19:30.301238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:30.301183    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:30.301344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:30.301293    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:30.301394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:30.301341    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:30.424005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:30.423968    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" event={"ID":"e4476629-9b97-427c-82df-d7b54567b2c8","Type":"ContainerStarted","Data":"c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6"}
Oct 30 13:19:30.424253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:30.424033    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" event={"ID":"e4476629-9b97-427c-82df-d7b54567b2c8","Type":"ContainerStarted","Data":"f16e8cb79ce066578e534bab93b9f9b734d06cedaf57eeb06408b9230d735772"}
Oct 30 13:19:30.477508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:30.473357    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" podStartSLOduration=1.473334762 podStartE2EDuration="1.473334762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:30.469720236 +0000 UTC m=+2834.945439840" watchObservedRunningTime="2026-10-30 13:19:30.473334762 +0000 UTC m=+2834.949054374"
Oct 30 13:19:30.886973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:30.886930    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:31.903030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:31.902998    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:32.446540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:32.446505    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:32.447477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:19:32.447458    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:19:38.570738 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:19:38.570695    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:20:08.619109 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:20:08.619065    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:20:11.801045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:11.801003    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"]
Oct 30 13:20:11.806177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:11.806144    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:11.820036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:11.820007    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"]
Oct 30 13:20:11.825245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:11.824081    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:20:11.839244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:11.836009    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:11.839244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:11.836119    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:20:11.839244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:11.836239    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:20:12.006636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.006589    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.006852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.006648    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7n9d\" (UniqueName: \"kubernetes.io/projected/e8d9950f-8e00-4499-b077-b1e0df88939e-kube-api-access-j7n9d\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.006852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.006773    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.107864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.107754    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.107864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.107810    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7n9d\" (UniqueName: \"kubernetes.io/projected/e8d9950f-8e00-4499-b077-b1e0df88939e-kube-api-access-j7n9d\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.108121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.107952    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.108777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.108749    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.110181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.110149    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.133841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.133807    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7n9d\" (UniqueName: \"kubernetes.io/projected/e8d9950f-8e00-4499-b077-b1e0df88939e-kube-api-access-j7n9d\") pod \"maas-api-revoke-keys-e2e-explicit-182377-ztq9v\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.140171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.140147    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:12.491254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.491002    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"]
Oct 30 13:20:12.831454 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:12.831354    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v" event={"ID":"e8d9950f-8e00-4499-b077-b1e0df88939e","Type":"ContainerStarted","Data":"908c539e23e953697c5a6e1f1677e671d5370dbdf95c633dfd8f072ef4b58875"}
Oct 30 13:20:13.448363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.448247    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"]
Oct 30 13:20:13.532024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.531988    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"]
Oct 30 13:20:13.532398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.532376    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.575184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.571728    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-037a5a\"/\"openshift-service-ca.crt\""
Oct 30 13:20:13.575184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.571881    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-037a5a\"/\"autores-model-037a5a-kserve-self-signed-certs\""
Oct 30 13:20:13.575184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.571928    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-037a5a\"/\"default-dockercfg-mmxd4\""
Oct 30 13:20:13.575184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.572023    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-037a5a\"/\"kube-root-ca.crt\""
Oct 30 13:20:13.631062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.631025    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tmp-dir\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.631245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.631075    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-model-cache\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.631245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.631150    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-dshm\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.631333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.631275    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tls-certs\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.631370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.631344    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8st\" (UniqueName: \"kubernetes.io/projected/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-kube-api-access-7z8st\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.631413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.631378    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-home\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.749947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.749907    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tmp-dir\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.750278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.750252    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-model-cache\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.750475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.750459    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-dshm\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.750673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.750659    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tls-certs\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.750836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.750820    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8st\" (UniqueName: \"kubernetes.io/projected/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-kube-api-access-7z8st\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.750943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.750930    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-home\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.751535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.751514    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-home\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.751921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.751901    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tmp-dir\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.752305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.752285    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-model-cache\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.784295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.778000    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-dshm\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.785336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.785310    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tls-certs\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.795247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.791023    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8st\" (UniqueName: \"kubernetes.io/projected/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-kube-api-access-7z8st\") pod \"autores-model-037a5a-kserve-7778d5cc97-z7qfl\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") " pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.852252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.851923    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:13.916544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.916240    4993 generic.go:356] "Generic (PLEG): container finished" podID="e8d9950f-8e00-4499-b077-b1e0df88939e" containerID="f640e096627611006477e12850a7fa4acc0280854b7dba470d9a6de7b71e7871" exitCode=0
Oct 30 13:20:13.916544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:13.916540    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v" event={"ID":"e8d9950f-8e00-4499-b077-b1e0df88939e","Type":"ContainerDied","Data":"f640e096627611006477e12850a7fa4acc0280854b7dba470d9a6de7b71e7871"}
Oct 30 13:20:14.286560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:14.286073    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"]
Oct 30 13:20:14.288654 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:20:14.288621    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc40b5492_a42d_4dd9_bf2b_4529e6fcf970.slice/crio-55949608a347382416d977e4d84ed5b16600e020fa99e14282cff67d977c988f WatchSource:0}: Error finding container 55949608a347382416d977e4d84ed5b16600e020fa99e14282cff67d977c988f: Status 404 returned error can't find the container with id 55949608a347382416d977e4d84ed5b16600e020fa99e14282cff67d977c988f
Oct 30 13:20:14.925194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:14.925150    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl" event={"ID":"c40b5492-a42d-4dd9-bf2b-4529e6fcf970","Type":"ContainerStarted","Data":"469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5"}
Oct 30 13:20:14.925194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:14.925199    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl" event={"ID":"c40b5492-a42d-4dd9-bf2b-4529e6fcf970","Type":"ContainerStarted","Data":"55949608a347382416d977e4d84ed5b16600e020fa99e14282cff67d977c988f"}
Oct 30 13:20:14.987234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:14.975333    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl" podStartSLOduration=1.975312371 podStartE2EDuration="1.975312371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:14.974623821 +0000 UTC m=+2879.450343431" watchObservedRunningTime="2026-10-30 13:20:14.975312371 +0000 UTC m=+2879.451031982"
Oct 30 13:20:15.198583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.198520    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:15.306409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.306378    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d9950f-8e00-4499-b077-b1e0df88939e-kube-api-access-j7n9d\" (UniqueName: \"kubernetes.io/projected/e8d9950f-8e00-4499-b077-b1e0df88939e-kube-api-access-j7n9d\") pod \"e8d9950f-8e00-4499-b077-b1e0df88939e\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") "
Oct 30 13:20:15.306606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.306433    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-openshift-ca\") pod \"e8d9950f-8e00-4499-b077-b1e0df88939e\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") "
Oct 30 13:20:15.306606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.306495    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-xks-ca\") pod \"e8d9950f-8e00-4499-b077-b1e0df88939e\" (UID: \"e8d9950f-8e00-4499-b077-b1e0df88939e\") "
Oct 30 13:20:15.306917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.306888    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-openshift-ca" pod "e8d9950f-8e00-4499-b077-b1e0df88939e" (UID: "e8d9950f-8e00-4499-b077-b1e0df88939e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:15.307102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.307086    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:15.308641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.308611    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-xks-ca" pod "e8d9950f-8e00-4499-b077-b1e0df88939e" (UID: "e8d9950f-8e00-4499-b077-b1e0df88939e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:15.308809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.308792    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d9950f-8e00-4499-b077-b1e0df88939e-kube-api-access-j7n9d" pod "e8d9950f-8e00-4499-b077-b1e0df88939e" (UID: "e8d9950f-8e00-4499-b077-b1e0df88939e"). InnerVolumeSpecName "kube-api-access-j7n9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:15.408333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.408299    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7n9d\" (UniqueName: \"kubernetes.io/projected/e8d9950f-8e00-4499-b077-b1e0df88939e-kube-api-access-j7n9d\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:15.408333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.408327    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8d9950f-8e00-4499-b077-b1e0df88939e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:15.853382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.853350    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:15.936590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.936548    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"
Oct 30 13:20:15.937080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.936583    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v" event={"ID":"e8d9950f-8e00-4499-b077-b1e0df88939e","Type":"ContainerDied","Data":"908c539e23e953697c5a6e1f1677e671d5370dbdf95c633dfd8f072ef4b58875"}
Oct 30 13:20:15.937080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:15.936647    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="908c539e23e953697c5a6e1f1677e671d5370dbdf95c633dfd8f072ef4b58875"
Oct 30 13:20:18.965017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:18.964984    4993 generic.go:356] "Generic (PLEG): container finished" podID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerID="445105b16cd4eefcd5e228ac2c8d77673656cfd8ad6447f65184bdde3aebcbc6" exitCode=0
Oct 30 13:20:18.965514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:18.965056    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" event={"ID":"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f","Type":"ContainerDied","Data":"445105b16cd4eefcd5e228ac2c8d77673656cfd8ad6447f65184bdde3aebcbc6"}
Oct 30 13:20:18.965578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:18.965557    4993 scope.go:122] "RemoveContainer" containerID="445105b16cd4eefcd5e228ac2c8d77673656cfd8ad6447f65184bdde3aebcbc6"
Oct 30 13:20:18.967591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:18.967553    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:18.967704 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:18.967650    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:18.967704 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:18.967695    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:19.991923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:19.991881    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" event={"ID":"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f","Type":"ContainerStarted","Data":"42338d650c6517db65f832080098669a437a373aae00d249974ad1373aba1d2c"}
Oct 30 13:20:20.793682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:20.793635    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:20:20.793907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:20.793878    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 13:20:20.793982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:20.793930    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 13:20:21.794286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:21.794245    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 13:20:21.794691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:21.794313    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 13:20:22.794834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:22.794346    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 13:20:22.794834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:22.794454    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 13:20:23.793943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:23.793905    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 13:20:23.794182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:23.793967    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 13:20:23.871233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:23.870269    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:23.897536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:23.897502    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:24.100777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:24.100687    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:24.794475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:24.794436    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 13:20:24.794700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:24.794493    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 13:20:25.798897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:25.798866    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:20:25.799393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:25.799373    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:20:25.800236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:25.800199    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:20:31.041135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:31.041096    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"]
Oct 30 13:20:31.065121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:31.065073    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f498539-7jt6n"]
Oct 30 13:20:31.691795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:31.691756    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23bda295-90f2-4710-b126-86a5271b5dc7" path="/var/lib/kubelet/pods/23bda295-90f2-4710-b126-86a5271b5dc7/volumes"
Oct 30 13:20:33.595826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.595783    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"]
Oct 30 13:20:33.597231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.597184    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e8d9950f-8e00-4499-b077-b1e0df88939e" containerName="revoke-keys"
Oct 30 13:20:33.597432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.597417    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d9950f-8e00-4499-b077-b1e0df88939e" containerName="revoke-keys"
Oct 30 13:20:33.597810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.597791    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d9950f-8e00-4499-b077-b1e0df88939e" containerName="revoke-keys"
Oct 30 13:20:33.628612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.622983    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"]
Oct 30 13:20:33.628612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.623127    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.712787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.661344    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2d2a0b-openshift-default-dockercfg-wcwc2\""
Oct 30 13:20:33.864479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.838654    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-token\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.879439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.879394    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-data\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.879778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.879757    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfhp\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-kube-api-access-jrfhp\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.880004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.879988    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-certs\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.883192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.880110    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.883635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.883603    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-credential-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.883794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.883778    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istiod-ca-cert\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.884019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.884000    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-podinfo\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.884173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.884140    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-envoy\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.990153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.990105    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-podinfo\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.990466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.990443    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-envoy\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.990661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.990646    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-token\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.990777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.990764    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-data\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.990955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.990941    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfhp\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-kube-api-access-jrfhp\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.991123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.991107    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-certs\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.991265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.991249    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.991473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.991457    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-credential-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.991579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.991563    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istiod-ca-cert\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:33.992568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:33.992541    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istiod-ca-cert\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.008464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.008429    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.008972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.008946    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-certs\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.009693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.009670    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-data\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.010064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.010045    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-credential-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.026271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.026236    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-envoy\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.041558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.040983    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfhp\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-kube-api-access-jrfhp\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.044819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.044753    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-token\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.057235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.053830    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-podinfo\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.246371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.246325    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:34.688194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.688079    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"]
Oct 30 13:20:34.689258 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:20:34.689232    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950a834d_9d0c_4b93_bcd6_1ef58e5b0bea.slice/crio-febf8726864f42130e0e2bd47636453cbebebd3bfd2079d19b75c068216d257a WatchSource:0}: Error finding container febf8726864f42130e0e2bd47636453cbebebd3bfd2079d19b75c068216d257a: Status 404 returned error can't find the container with id febf8726864f42130e0e2bd47636453cbebebd3bfd2079d19b75c068216d257a
Oct 30 13:20:34.692150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.692111    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:34.692293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.692183    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:34.692293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:34.692228    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:35.216943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:35.216894    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" event={"ID":"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea","Type":"ContainerStarted","Data":"7619a3a4b739d7972eb1429f3403df469040cece3176b68783607b9428e205a8"}
Oct 30 13:20:35.217172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:35.216968    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" event={"ID":"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea","Type":"ContainerStarted","Data":"febf8726864f42130e0e2bd47636453cbebebd3bfd2079d19b75c068216d257a"}
Oct 30 13:20:35.247080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:35.247033    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:35.248376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:35.248347    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:20:35.248522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:35.248398    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 13:20:35.274231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:35.273973    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podStartSLOduration=2.2739517559999998 podStartE2EDuration="2.273951756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:35.268745404 +0000 UTC m=+2899.744465015" watchObservedRunningTime="2026-10-30 13:20:35.273951756 +0000 UTC m=+2899.749671369"
Oct 30 13:20:36.252645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:36.252611    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:37.238367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:37.238330    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:37.239534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:37.239504    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:20:38.571541 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:20:38.571496    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:20:38.993972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:38.993927    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"]
Oct 30 13:20:39.771515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.771473    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"]
Oct 30 13:20:39.777539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.777515    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:39.794914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.794881    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:39.797584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.795530    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:20:39.797584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.795580    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:20:39.797584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.795676    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:20:39.812338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.802008    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"]
Oct 30 13:20:39.910554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.910361    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:39.910668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.910554    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlct8\" (UniqueName: \"kubernetes.io/projected/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-kube-api-access-zlct8\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:39.910736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:39.910684    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:40.012842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.012796    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:40.013059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.012932    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:40.013133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.013057    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlct8\" (UniqueName: \"kubernetes.io/projected/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-kube-api-access-zlct8\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:40.013664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.013639    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:40.015959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.015930    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:40.039009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.038937    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlct8\" (UniqueName: \"kubernetes.io/projected/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-kube-api-access-zlct8\") pod \"maas-api-revoke-keys-e2e-autores-037a5a-nlvkk\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:40.100613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.100520    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:40.280538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.277733    4993 generic.go:356] "Generic (PLEG): container finished" podID="e4476629-9b97-427c-82df-d7b54567b2c8" containerID="c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6" exitCode=0
Oct 30 13:20:40.280538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.277868    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" event={"ID":"e4476629-9b97-427c-82df-d7b54567b2c8","Type":"ContainerDied","Data":"c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6"}
Oct 30 13:20:40.280538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.278382    4993 scope.go:122] "RemoveContainer" containerID="c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6"
Oct 30 13:20:40.284982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.284695    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:40.284982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.284804    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:40.284982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.284847    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:40.572951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:40.572860    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"]
Oct 30 13:20:40.576589 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:20:40.576563    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b11c0dc_803e_4799_ba1f_d74fed9ae43a.slice/crio-97242232d909cd69b64eccff4c0cd3172440878c6e70672fd855ae814a714550 WatchSource:0}: Error finding container 97242232d909cd69b64eccff4c0cd3172440878c6e70672fd855ae814a714550: Status 404 returned error can't find the container with id 97242232d909cd69b64eccff4c0cd3172440878c6e70672fd855ae814a714550
Oct 30 13:20:41.299928 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:41.299821    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk" event={"ID":"0b11c0dc-803e-4799-ba1f-d74fed9ae43a","Type":"ContainerStarted","Data":"52a940c8994546871fd5cc34bd68dd99f444edbe0c52d52f75ae46a451a811aa"}
Oct 30 13:20:41.299928 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:41.299887    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk" event={"ID":"0b11c0dc-803e-4799-ba1f-d74fed9ae43a","Type":"ContainerStarted","Data":"97242232d909cd69b64eccff4c0cd3172440878c6e70672fd855ae814a714550"}
Oct 30 13:20:41.302615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:41.302582    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" event={"ID":"e4476629-9b97-427c-82df-d7b54567b2c8","Type":"ContainerStarted","Data":"e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019"}
Oct 30 13:20:41.344782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:41.344724    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk" podStartSLOduration=2.34470917 podStartE2EDuration="2.34470917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:41.341511941 +0000 UTC m=+2905.817231552" watchObservedRunningTime="2026-10-30 13:20:41.34470917 +0000 UTC m=+2905.820428780"
Oct 30 13:20:41.887122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:41.887083    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:20:41.887441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:41.887404    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 13:20:41.887520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:41.887462    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 13:20:42.312081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:42.312040    4993 generic.go:356] "Generic (PLEG): container finished" podID="0b11c0dc-803e-4799-ba1f-d74fed9ae43a" containerID="52a940c8994546871fd5cc34bd68dd99f444edbe0c52d52f75ae46a451a811aa" exitCode=0
Oct 30 13:20:42.312529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:42.312120    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk" event={"ID":"0b11c0dc-803e-4799-ba1f-d74fed9ae43a","Type":"ContainerDied","Data":"52a940c8994546871fd5cc34bd68dd99f444edbe0c52d52f75ae46a451a811aa"}
Oct 30 13:20:42.887877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:42.887825    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 13:20:42.888107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:42.887898    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 13:20:43.628043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.628014    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:43.729158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.729118    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-kube-api-access-zlct8\" (UniqueName: \"kubernetes.io/projected/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-kube-api-access-zlct8\") pod \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") "
Oct 30 13:20:43.729473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.729437    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-openshift-ca\") pod \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") "
Oct 30 13:20:43.729587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.729487    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-xks-ca\") pod \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\" (UID: \"0b11c0dc-803e-4799-ba1f-d74fed9ae43a\") "
Oct 30 13:20:43.731189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.731150    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-openshift-ca" pod "0b11c0dc-803e-4799-ba1f-d74fed9ae43a" (UID: "0b11c0dc-803e-4799-ba1f-d74fed9ae43a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:43.732736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.732698    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-xks-ca" pod "0b11c0dc-803e-4799-ba1f-d74fed9ae43a" (UID: "0b11c0dc-803e-4799-ba1f-d74fed9ae43a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:43.732991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.732963    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-kube-api-access-zlct8" pod "0b11c0dc-803e-4799-ba1f-d74fed9ae43a" (UID: "0b11c0dc-803e-4799-ba1f-d74fed9ae43a"). InnerVolumeSpecName "kube-api-access-zlct8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:43.831903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.831865    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.831903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.831900    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.831903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.831910    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlct8\" (UniqueName: \"kubernetes.io/projected/0b11c0dc-803e-4799-ba1f-d74fed9ae43a-kube-api-access-zlct8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.888235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.888060    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 13:20:43.888235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:43.888171    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 13:20:44.331865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:44.331834    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"
Oct 30 13:20:44.332151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:44.331853    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk" event={"ID":"0b11c0dc-803e-4799-ba1f-d74fed9ae43a","Type":"ContainerDied","Data":"97242232d909cd69b64eccff4c0cd3172440878c6e70672fd855ae814a714550"}
Oct 30 13:20:44.332289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:44.332177    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97242232d909cd69b64eccff4c0cd3172440878c6e70672fd855ae814a714550"
Oct 30 13:20:44.888232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:44.887796    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 13:20:44.888232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:44.887872    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 13:20:45.886922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:45.886873    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 13:20:45.887126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:45.886936    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 13:20:47.053517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:47.053466    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:20:47.054050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:47.053992    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:20:47.055999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:47.055969    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:20:54.026315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.026264    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl" podUID="c40b5492-a42d-4dd9-bf2b-4529e6fcf970" containerName="main" containerID="cri-o://469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5" gracePeriod=45
Oct 30 13:20:54.088548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.088508    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl" podUID="c40b5492-a42d-4dd9-bf2b-4529e6fcf970" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": dial tcp 10.132.0.216:8000: connect: connection refused"
Oct 30 13:20:54.336779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.336744    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:54.421813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.421772    4993 generic.go:356] "Generic (PLEG): container finished" podID="c40b5492-a42d-4dd9-bf2b-4529e6fcf970" containerID="469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5" exitCode=0
Oct 30 13:20:54.422035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.421824    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl" event={"ID":"c40b5492-a42d-4dd9-bf2b-4529e6fcf970","Type":"ContainerDied","Data":"469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5"}
Oct 30 13:20:54.422035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.421838    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"
Oct 30 13:20:54.422035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.421863    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl" event={"ID":"c40b5492-a42d-4dd9-bf2b-4529e6fcf970","Type":"ContainerDied","Data":"55949608a347382416d977e4d84ed5b16600e020fa99e14282cff67d977c988f"}
Oct 30 13:20:54.422035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.421885    4993 scope.go:122] "RemoveContainer" containerID="469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5"
Oct 30 13:20:54.433834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.433798    4993 scope.go:122] "RemoveContainer" containerID="469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5"
Oct 30 13:20:54.434264 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:20:54.434236    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5\": container with ID starting with 469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5 not found: ID does not exist" containerID="469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5"
Oct 30 13:20:54.434378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.434271    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5"} err="failed to get container status \"469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5\": rpc error: code = NotFound desc = could not find container \"469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5\": container with ID starting with 469194a7d87e76a91943bec2378f86d51f58cc719dcd85a3001ea52348be7de5 not found: ID does not exist"
Oct 30 13:20:54.469511 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.469471    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tls-certs\" (UniqueName: \"kubernetes.io/secret/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tls-certs\") pod \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") "
Oct 30 13:20:54.469716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.469564    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-kube-api-access-7z8st\" (UniqueName: \"kubernetes.io/projected/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-kube-api-access-7z8st\") pod \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") "
Oct 30 13:20:54.469716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.469634    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-model-cache\") pod \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") "
Oct 30 13:20:54.469716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.469697    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tmp-dir\") pod \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") "
Oct 30 13:20:54.469885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.469761    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-home\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-home\") pod \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") "
Oct 30 13:20:54.469885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.469824    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-dshm\") pod \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\" (UID: \"c40b5492-a42d-4dd9-bf2b-4529e6fcf970\") "
Oct 30 13:20:54.470027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.469997    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tmp-dir" pod "c40b5492-a42d-4dd9-bf2b-4529e6fcf970" (UID: "c40b5492-a42d-4dd9-bf2b-4529e6fcf970"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:54.470085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.470039    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-home" pod "c40b5492-a42d-4dd9-bf2b-4529e6fcf970" (UID: "c40b5492-a42d-4dd9-bf2b-4529e6fcf970"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:54.470455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.470431    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:54.470552 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.470460    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:54.470666 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.470644    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-model-cache" pod "c40b5492-a42d-4dd9-bf2b-4529e6fcf970" (UID: "c40b5492-a42d-4dd9-bf2b-4529e6fcf970"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:54.472274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.472241    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-kube-api-access-7z8st" pod "c40b5492-a42d-4dd9-bf2b-4529e6fcf970" (UID: "c40b5492-a42d-4dd9-bf2b-4529e6fcf970"). InnerVolumeSpecName "kube-api-access-7z8st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:54.472414 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.472287    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tls-certs" pod "c40b5492-a42d-4dd9-bf2b-4529e6fcf970" (UID: "c40b5492-a42d-4dd9-bf2b-4529e6fcf970"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:54.472761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.472737    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-dshm" pod "c40b5492-a42d-4dd9-bf2b-4529e6fcf970" (UID: "c40b5492-a42d-4dd9-bf2b-4529e6fcf970"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:54.572561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.572472    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:54.572561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.572513    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:54.572561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.572528    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z8st\" (UniqueName: \"kubernetes.io/projected/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-kube-api-access-7z8st\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:54.572561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.572542    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c40b5492-a42d-4dd9-bf2b-4529e6fcf970-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:20:54.770761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.770716    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"]
Oct 30 13:20:54.782366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:54.782327    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-037a5a/autores-model-037a5a-kserve-7778d5cc97-z7qfl"]
Oct 30 13:20:55.697879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:55.697253    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40b5492-a42d-4dd9-bf2b-4529e6fcf970" path="/var/lib/kubelet/pods/c40b5492-a42d-4dd9-bf2b-4529e6fcf970/volumes"
Oct 30 13:20:56.394515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:56.394421    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"]
Oct 30 13:20:56.405896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:56.405855    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-182377-ztq9v"]
Oct 30 13:20:57.714313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:20:57.714271    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d9950f-8e00-4499-b077-b1e0df88939e" path="/var/lib/kubelet/pods/e8d9950f-8e00-4499-b077-b1e0df88939e/volumes"
Oct 30 13:21:08.573653 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:21:08.573609    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:21:15.918793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.918737    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"]
Oct 30 13:21:15.919664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.919637    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0b11c0dc-803e-4799-ba1f-d74fed9ae43a" containerName="revoke-keys"
Oct 30 13:21:15.919783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.919666    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b11c0dc-803e-4799-ba1f-d74fed9ae43a" containerName="revoke-keys"
Oct 30 13:21:15.919783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.919696    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c40b5492-a42d-4dd9-bf2b-4529e6fcf970" containerName="main"
Oct 30 13:21:15.919783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.919704    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40b5492-a42d-4dd9-bf2b-4529e6fcf970" containerName="main"
Oct 30 13:21:15.919961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.919898    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40b5492-a42d-4dd9-bf2b-4529e6fcf970" containerName="main"
Oct 30 13:21:15.919961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.919925    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b11c0dc-803e-4799-ba1f-d74fed9ae43a" containerName="revoke-keys"
Oct 30 13:21:15.923117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.923088    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:15.938074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.938042    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-75d385-openshift-default-dockercfg-fhdv5\""
Oct 30 13:21:15.947722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:15.947685    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"]
Oct 30 13:21:16.145097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.145047    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.145422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.145397    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-certs\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.145573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.145556    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-credential-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.145693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.145678    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-istio-token\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.145817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.145803    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3315757c-d2b9-4025-adb4-1eb495aef420-istiod-ca-cert\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.145943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.145929    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-envoy\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.146149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.146134    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-data\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.146332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.146316    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3315757c-d2b9-4025-adb4-1eb495aef420-istio-podinfo\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.146558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.146542    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntp8\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-kube-api-access-7ntp8\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.249294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.249253    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntp8\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-kube-api-access-7ntp8\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.249616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.249591    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.249804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.249787    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-certs\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.250270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.250251    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-credential-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.250724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.250708    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-istio-token\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.251113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.251096    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3315757c-d2b9-4025-adb4-1eb495aef420-istiod-ca-cert\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.253584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.253563    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-envoy\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.253826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.253811    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-data\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.254701 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.254683    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3315757c-d2b9-4025-adb4-1eb495aef420-istio-podinfo\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.255381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.253332    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3315757c-d2b9-4025-adb4-1eb495aef420-istiod-ca-cert\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.255539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.250657    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-credential-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.255636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.254572    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-data\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.255724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.250184    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-certs\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.255814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.250183    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.282715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.282670    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-istio-token\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.285245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.283616    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntp8\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-kube-api-access-7ntp8\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.291993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.291962    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3315757c-d2b9-4025-adb4-1eb495aef420-istio-podinfo\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.307276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.305519    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-envoy\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.538075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.537972    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:16.916976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.916941    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"]
Oct 30 13:21:16.919824 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:21:16.919787    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3315757c_d2b9_4025_adb4_1eb495aef420.slice/crio-525c1935cca9d867504d6ffe58d385d9a08f86573996557ee81e50822df37ab8 WatchSource:0}: Error finding container 525c1935cca9d867504d6ffe58d385d9a08f86573996557ee81e50822df37ab8: Status 404 returned error can't find the container with id 525c1935cca9d867504d6ffe58d385d9a08f86573996557ee81e50822df37ab8
Oct 30 13:21:16.922709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.922672    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:16.922817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.922742    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:16.922817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:16.922771    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:17.646005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:17.645967    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" event={"ID":"3315757c-d2b9-4025-adb4-1eb495aef420","Type":"ContainerStarted","Data":"175e46f7819ed55164201e0d734a1e0ccc3ce6eaea6334f08bc19aba60b84612"}
Oct 30 13:21:17.646239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:17.646018    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" event={"ID":"3315757c-d2b9-4025-adb4-1eb495aef420","Type":"ContainerStarted","Data":"525c1935cca9d867504d6ffe58d385d9a08f86573996557ee81e50822df37ab8"}
Oct 30 13:21:17.717682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:17.717627    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" podStartSLOduration=2.7176107739999997 podStartE2EDuration="2.717610774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:17.703169166 +0000 UTC m=+2942.178888775" watchObservedRunningTime="2026-10-30 13:21:17.717610774 +0000 UTC m=+2942.193330391"
Oct 30 13:21:18.539202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:18.539166    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:18.545484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:18.545455    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:18.653820 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:18.653784    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:18.654893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:18.654865    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:21:22.977577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:22.977540    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"]
Oct 30 13:21:22.981021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:22.980996    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-037a5a-nlvkk"]
Oct 30 13:21:23.713495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:23.713463    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b11c0dc-803e-4799-ba1f-d74fed9ae43a" path="/var/lib/kubelet/pods/0b11c0dc-803e-4799-ba1f-d74fed9ae43a/volumes"
Oct 30 13:21:38.569367 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:21:38.569328    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:21:51.002319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:51.002284    4993 generic.go:356] "Generic (PLEG): container finished" podID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerID="7619a3a4b739d7972eb1429f3403df469040cece3176b68783607b9428e205a8" exitCode=0
Oct 30 13:21:51.002902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:51.002360    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" event={"ID":"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea","Type":"ContainerDied","Data":"7619a3a4b739d7972eb1429f3403df469040cece3176b68783607b9428e205a8"}
Oct 30 13:21:51.002902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:51.002868    4993 scope.go:122] "RemoveContainer" containerID="7619a3a4b739d7972eb1429f3403df469040cece3176b68783607b9428e205a8"
Oct 30 13:21:51.004752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:51.004720    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:51.004851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:51.004797    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:51.004851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:51.004829    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:52.021539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:52.020246    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" event={"ID":"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea","Type":"ContainerStarted","Data":"7ed52b79d78d5e0eb7ea68c48f4efa8f748d8a56b4972d34cbe4d29e241053c1"}
Oct 30 13:21:52.238497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:52.238455    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:21:52.247692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:52.247667    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:21:52.249194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:52.249161    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:21:52.249357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:52.249228    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 13:21:53.247925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:53.247883    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:21:53.248421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:53.247945    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 13:21:54.247691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:54.247652    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:21:54.247879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:54.247710    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 13:21:55.247501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:55.247462    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:21:55.247903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:55.247519    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 13:21:56.251031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:56.250993    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:21:56.251691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:56.251660    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 13:21:57.247708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:57.247665    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:21:57.247946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:57.247749    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 30 13:21:58.252680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:58.252642    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:21:58.254008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:21:58.253986    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:22:08.568618 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:22:08.568571    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:22:16.226781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:16.226751    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:22:16.252762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:16.252730    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:22:19.614518 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:19.614485    4993 scope.go:122] "RemoveContainer" containerID="654916bf2be17a13827d02012f344a1f17154f3a885c95d1d968d264dc84ce86"
Oct 30 13:22:27.576459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:27.576371    4993 generic.go:356] "Generic (PLEG): container finished" podID="3315757c-d2b9-4025-adb4-1eb495aef420" containerID="175e46f7819ed55164201e0d734a1e0ccc3ce6eaea6334f08bc19aba60b84612" exitCode=0
Oct 30 13:22:27.576459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:27.576427    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" event={"ID":"3315757c-d2b9-4025-adb4-1eb495aef420","Type":"ContainerDied","Data":"175e46f7819ed55164201e0d734a1e0ccc3ce6eaea6334f08bc19aba60b84612"}
Oct 30 13:22:27.577078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:27.576988    4993 scope.go:122] "RemoveContainer" containerID="175e46f7819ed55164201e0d734a1e0ccc3ce6eaea6334f08bc19aba60b84612"
Oct 30 13:22:27.579053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:27.579017    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:27.579192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:27.579092    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:27.579192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:27.579133    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:28.594578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:28.594540    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" event={"ID":"3315757c-d2b9-4025-adb4-1eb495aef420","Type":"ContainerStarted","Data":"42851da24d83b04cab294bcb1f4173bf18db947879ba216812821825de98934c"}
Oct 30 13:22:29.538748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:29.538708    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:22:29.540256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:29.540195    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:22:29.540397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:29.540280    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 13:22:30.539501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:30.539456    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:22:30.539941 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:30.539518    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 13:22:31.539469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:31.539424    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:22:31.539687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:31.539491    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 13:22:32.538887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:32.538835    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:22:32.539107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:32.538905    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 13:22:33.046299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:33.046257    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"]
Oct 30 13:22:33.079783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:33.079742    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38547e1c-4scdd"]
Oct 30 13:22:33.539043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:33.539007    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:22:33.539362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:33.539329    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 13:22:33.676232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:33.675313    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:22:33.737990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:33.737926    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1" path="/var/lib/kubelet/pods/c425e1cc-f94d-4b39-a1c6-6e6aa52e94c1/volumes"
Oct 30 13:22:34.544487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:34.544457    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:22:34.642716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:34.642683    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:22:38.571316 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:22:38.571273    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:22:39.721013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.720973    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"]
Oct 30 13:22:39.726030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.725974    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.744081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.744056    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-88e9f3-openshift-default-dockercfg-f4tkz\""
Oct 30 13:22:39.765115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.765079    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"]
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874030    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istiod-ca-cert\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874172    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-certs\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874221    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-envoy\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874374    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-credential-socket\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874412    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-token\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874450    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-data\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874481    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbjz\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-kube-api-access-wtbjz\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874561    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-podinfo\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.876332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.874623    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-socket\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.980199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.980157    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istiod-ca-cert\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.982434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.982407    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-certs\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.983101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.983055    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-envoy\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.983472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.983446    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-credential-socket\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.983674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.983644    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-token\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.983863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.983838    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-data\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.983997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.983983    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbjz\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-kube-api-access-wtbjz\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.984192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.984177    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-podinfo\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.984396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.984379    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-socket\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.984512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.984484    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-data\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.984591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.982235    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istiod-ca-cert\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.984591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.983912    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-credential-socket\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:39.984591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.982984    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-certs\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:40.002307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:39.985071    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-socket\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:40.013235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.009853    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-envoy\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:40.013235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.012481    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-podinfo\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:40.040632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.040328    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-token\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:40.063227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.063140    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbjz\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-kube-api-access-wtbjz\") pod \"e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") " pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:40.344899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.344795    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:40.703598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.703498    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"]
Oct 30 13:22:40.710706 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:22:40.710667    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e3b0a0_6454_4b4e_bc88_9d4869ea0ed9.slice/crio-4016ad9d213578095d51bb68aa0adf59961d1910dfd31facf782fc30ea87ee81 WatchSource:0}: Error finding container 4016ad9d213578095d51bb68aa0adf59961d1910dfd31facf782fc30ea87ee81: Status 404 returned error can't find the container with id 4016ad9d213578095d51bb68aa0adf59961d1910dfd31facf782fc30ea87ee81
Oct 30 13:22:40.714616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.714568    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:40.714722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.714668    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:40.714765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:40.714716    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:41.717670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:41.717630    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" event={"ID":"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9","Type":"ContainerStarted","Data":"13ddf3162ee0f030f1d304ba9802236862261cbe9e0f361ab2ba7d6f284dd9b6"}
Oct 30 13:22:41.718101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:41.717689    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" event={"ID":"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9","Type":"ContainerStarted","Data":"4016ad9d213578095d51bb68aa0adf59961d1910dfd31facf782fc30ea87ee81"}
Oct 30 13:22:41.787190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:41.787127    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" podStartSLOduration=2.787107657 podStartE2EDuration="2.787107657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:41.775446247 +0000 UTC m=+3026.251165839" watchObservedRunningTime="2026-10-30 13:22:41.787107657 +0000 UTC m=+3026.262827267"
Oct 30 13:22:42.345201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:42.345167    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:42.354879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:42.354849    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:42.727142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:42.727055    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:42.728616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:42.728587    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:22:43.031252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:43.031128    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"]
Oct 30 13:22:43.034940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:43.034904    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b231f234-v6t57"]
Oct 30 13:22:43.692365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:43.692329    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d93c23d-75fc-432e-bf15-0eab1b8ef2c7" path="/var/lib/kubelet/pods/5d93c23d-75fc-432e-bf15-0eab1b8ef2c7/volumes"
Oct 30 13:22:55.044883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:55.044847    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"]
Oct 30 13:22:55.050959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:55.050924    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ae6f37-zh5rr"]
Oct 30 13:22:55.692194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:22:55.692156    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3942337b-3aad-46dc-a969-2390b55cea83" path="/var/lib/kubelet/pods/3942337b-3aad-46dc-a969-2390b55cea83/volumes"
Oct 30 13:23:05.849290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.849249    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"]
Oct 30 13:23:05.861243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.857848    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.881948 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.881911    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"]
Oct 30 13:23:05.888025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.887984    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0d9115-openshift-default-dockercfg-7m5k5\""
Oct 30 13:23:05.959230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.959163    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.959473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.959270    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-envoy\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.959473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.959367    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-certs\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.990441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.990384    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-token\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.990852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.990822    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28kz\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-kube-api-access-h28kz\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.991022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.991005    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istiod-ca-cert\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.991172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.991153    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-podinfo\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.991376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.991353    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-credential-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:05.991594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:05.991571    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-data\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.104782    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-token\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.104873    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28kz\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-kube-api-access-h28kz\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.104938    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istiod-ca-cert\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.104992    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-podinfo\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.105065    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-credential-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.105104    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-data\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.105197    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.105269    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-envoy\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.105364    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-certs\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.105710    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-certs\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.106172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.106082    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istiod-ca-cert\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.107781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.107053    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-data\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.107781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.107426    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.123675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.120276    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"]
Oct 30 13:23:06.123675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.120425    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-credential-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.148243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.148173    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ae558-vjhtz"]
Oct 30 13:23:06.151979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.151897    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-podinfo\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.161131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.161045    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28kz\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-kube-api-access-h28kz\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.164247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.163813    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-token\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.164247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.164016    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-envoy\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.179708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.179613    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:06.628174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.628115    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"]
Oct 30 13:23:06.631156 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:23:06.631117    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5ff6f0_dce9_4419_aa9e_abe3045212f7.slice/crio-50a6712885b75d7c74dee0ebc03da520f47b5bafce5fb33fbf81d9b82f56eda5 WatchSource:0}: Error finding container 50a6712885b75d7c74dee0ebc03da520f47b5bafce5fb33fbf81d9b82f56eda5: Status 404 returned error can't find the container with id 50a6712885b75d7c74dee0ebc03da520f47b5bafce5fb33fbf81d9b82f56eda5
Oct 30 13:23:06.634166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.634085    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:06.634359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.634175    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:06.634359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:06.634238    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:07.119295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:07.119253    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" event={"ID":"7b5ff6f0-dce9-4419-aa9e-abe3045212f7","Type":"ContainerStarted","Data":"1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd"}
Oct 30 13:23:07.119849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:07.119314    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" event={"ID":"7b5ff6f0-dce9-4419-aa9e-abe3045212f7","Type":"ContainerStarted","Data":"50a6712885b75d7c74dee0ebc03da520f47b5bafce5fb33fbf81d9b82f56eda5"}
Oct 30 13:23:07.180525 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:07.180487    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:07.182334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:07.182273    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" podStartSLOduration=2.182254 podStartE2EDuration="2.182254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:07.176805689 +0000 UTC m=+3051.652525299" watchObservedRunningTime="2026-10-30 13:23:07.182254 +0000 UTC m=+3051.657973614"
Oct 30 13:23:07.187184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:07.187155    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:07.692626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:07.692589    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c661b0b0-3112-4103-8726-a9c393b027e2" path="/var/lib/kubelet/pods/c661b0b0-3112-4103-8726-a9c393b027e2/volumes"
Oct 30 13:23:08.130750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:08.130717    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:08.132494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:08.132466    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:23:08.570084 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:23:08.570045    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:23:19.642215 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:19.642165    4993 scope.go:122] "RemoveContainer" containerID="65605793c67f5e4050bcec70fc903719ec266470e8b628c4a54122feb8b0db0c"
Oct 30 13:23:24.031731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:24.031687    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"]
Oct 30 13:23:24.041244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:24.041199    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b05e99a-6vj24"]
Oct 30 13:23:25.696114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:25.696071    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d70f700-0873-4ce2-a36c-272859f50ab8" path="/var/lib/kubelet/pods/3d70f700-0873-4ce2-a36c-272859f50ab8/volumes"
Oct 30 13:23:27.038272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:27.038232    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"]
Oct 30 13:23:27.051909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:27.051873    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099b5205-l8r8d"]
Oct 30 13:23:27.776271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:27.776194    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2138d10-0883-4a24-b6d5-d1bc847fc8dc" path="/var/lib/kubelet/pods/c2138d10-0883-4a24-b6d5-d1bc847fc8dc/volumes"
Oct 30 13:23:35.031634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:35.031591    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"]
Oct 30 13:23:35.043359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:35.043325    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593b3a2a-fjpr8"]
Oct 30 13:23:35.692760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:35.692721    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b6e425-c818-4bad-aee1-84118551bfeb" path="/var/lib/kubelet/pods/f2b6e425-c818-4bad-aee1-84118551bfeb/volumes"
Oct 30 13:23:38.029136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:38.029094    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"]
Oct 30 13:23:38.041599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:38.041560    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae602ca-xr7hg"]
Oct 30 13:23:38.571896 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:23:38.571844    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:23:39.695959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:39.695926    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adc327e-c516-47ba-ad9f-6bd584d28610" path="/var/lib/kubelet/pods/6adc327e-c516-47ba-ad9f-6bd584d28610/volumes"
Oct 30 13:23:54.625288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:54.625180    4993 generic.go:356] "Generic (PLEG): container finished" podID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerID="13ddf3162ee0f030f1d304ba9802236862261cbe9e0f361ab2ba7d6f284dd9b6" exitCode=0
Oct 30 13:23:54.625288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:54.625240    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" event={"ID":"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9","Type":"ContainerDied","Data":"13ddf3162ee0f030f1d304ba9802236862261cbe9e0f361ab2ba7d6f284dd9b6"}
Oct 30 13:23:54.625866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:54.625679    4993 scope.go:122] "RemoveContainer" containerID="13ddf3162ee0f030f1d304ba9802236862261cbe9e0f361ab2ba7d6f284dd9b6"
Oct 30 13:23:54.627579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:54.627547    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:54.627687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:54.627622    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:54.627687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:54.627665    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:55.635727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:55.635690    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" event={"ID":"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9","Type":"ContainerStarted","Data":"7a82797ed5dd41b241b84079f3c334b508016c99de869b2df5aab21ed7a9cdbb"}
Oct 30 13:23:56.345781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:56.345697    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:23:56.347139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:56.347113    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 13:23:56.347284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:56.347170    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 13:23:57.345311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:57.345276    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 13:23:57.345766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:57.345330    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 13:23:57.727085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:57.727050    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:23:58.345897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:58.345851    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 13:23:58.346332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:58.345927    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 13:23:59.345657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:59.345613    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 13:23:59.345835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:23:59.345671    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 13:24:00.345312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:00.345274    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 13:24:00.345836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:00.345798    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 13:24:01.350927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:01.350896    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:24:01.352170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:01.352145    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:24:08.571362 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:24:08.571320    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:24:14.049974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:14.049935    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"]
Oct 30 13:24:14.050440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:14.050175    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy" containerID="cri-o://98c06f3aeeb489af0da86c21bf2ab57e6614d083ede766b0458c4815a659de8f" gracePeriod=30
Oct 30 13:24:15.696273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:15.696236    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"]
Oct 30 13:24:15.697058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:15.697025    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy" containerID="cri-o://174bda2ab6ac29a784a66e8657d23d6b39173c73a8cf3993c1c51c488a2a1ef2" gracePeriod=30
Oct 30 13:24:17.244478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:17.244437    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:24:17.244902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:17.244524    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:24:19.331540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.331465    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 13:24:19.438541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.438506    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-envoy\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.438772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.438754    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-credential-socket\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.438871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.438830    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c482043e-237d-4780-95eb-ee9dedf162bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482043e-237d-4780-95eb-ee9dedf162bc-istiod-ca-cert\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.439041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.438934    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c482043e-237d-4780-95eb-ee9dedf162bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482043e-237d-4780-95eb-ee9dedf162bc-istio-podinfo\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.439041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.438992    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-istio-token\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-istio-token\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.439446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.439421    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-socket\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.439688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.439666    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-kube-api-access-wgttz\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-kube-api-access-wgttz\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.439964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.439700    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c482043e-237d-4780-95eb-ee9dedf162bc-istiod-ca-cert" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:19.439964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.439837    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-credential-socket" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.440152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.439984    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-data\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.440152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440023    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-socket" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.440152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440064    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-certs\") pod \"c482043e-237d-4780-95eb-ee9dedf162bc\" (UID: \"c482043e-237d-4780-95eb-ee9dedf162bc\") "
Oct 30 13:24:19.440431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440273    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-data" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.440498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440449    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-certs" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.440702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440686    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.440785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440704    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.440785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440713    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.440785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440721    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.440785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.440729    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482043e-237d-4780-95eb-ee9dedf162bc-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.441635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.441611    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-envoy" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.442464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.442430    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-kube-api-access-wgttz" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "kube-api-access-wgttz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:19.447381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.447347    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-istio-token" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:19.477271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.469047    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c482043e-237d-4780-95eb-ee9dedf162bc-istio-podinfo" pod "c482043e-237d-4780-95eb-ee9dedf162bc" (UID: "c482043e-237d-4780-95eb-ee9dedf162bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:19.480749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.480010    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:24:19.480749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.480349    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:24:19.542031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.541962    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgttz\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-kube-api-access-wgttz\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.542031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.542018    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482043e-237d-4780-95eb-ee9dedf162bc-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.542031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.542031    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482043e-237d-4780-95eb-ee9dedf162bc-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.542560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.542043    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482043e-237d-4780-95eb-ee9dedf162bc-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.745232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.732657    4993 scope.go:122] "RemoveContainer" containerID="98c06f3aeeb489af0da86c21bf2ab57e6614d083ede766b0458c4815a659de8f"
Oct 30 13:24:19.779669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.779263    4993 scope.go:122] "RemoveContainer" containerID="26951efafc19af710f7ad4b54f173ddbbaecdd4e071c8b91895fe09e1c3adb45"
Oct 30 13:24:19.824446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.824033    4993 scope.go:122] "RemoveContainer" containerID="b7565e303ed717477edebcb9a825682572df9c40db2bb4e0b329603d78167d6c"
Oct 30 13:24:19.880157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.879656    4993 scope.go:122] "RemoveContainer" containerID="f34ad4abdf4684b6b44de82674ecfd95e73fe18916aeeec4f3085764924ce953"
Oct 30 13:24:19.923116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.919570    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" event={"ID":"c482043e-237d-4780-95eb-ee9dedf162bc","Type":"ContainerDied","Data":"98c06f3aeeb489af0da86c21bf2ab57e6614d083ede766b0458c4815a659de8f"}
Oct 30 13:24:19.923116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.919645    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl" event={"ID":"c482043e-237d-4780-95eb-ee9dedf162bc","Type":"ContainerDied","Data":"492434dc951b3dc641802893fdb619cef7a7de591798797be2dafe57e45ec6d4"}
Oct 30 13:24:19.923116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.919731    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"
Oct 30 13:24:19.970274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.970249    4993 scope.go:122] "RemoveContainer" containerID="3f0523bfb167385bf6db3b6a056ecbc7b2a7b58a43bcdc2ae6a0321490bf5654"
Oct 30 13:24:19.989685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:19.989639    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"]
Oct 30 13:24:20.026368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:20.022354    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-8c4jl"]
Oct 30 13:24:21.015096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.015005    4993 generic.go:356] "Generic (PLEG): container finished" podID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerID="174bda2ab6ac29a784a66e8657d23d6b39173c73a8cf3993c1c51c488a2a1ef2" exitCode=0
Oct 30 13:24:21.015795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.015763    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" event={"ID":"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b","Type":"ContainerDied","Data":"174bda2ab6ac29a784a66e8657d23d6b39173c73a8cf3993c1c51c488a2a1ef2"}
Oct 30 13:24:21.059382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.059331    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"]
Oct 30 13:24:21.060560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.060529    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy"
Oct 30 13:24:21.060743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.060727    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy"
Oct 30 13:24:21.061067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.061048    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" containerName="istio-proxy"
Oct 30 13:24:21.064625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.064603    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.082284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.079558    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"routing-overlay\""
Oct 30 13:24:21.082284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.079673    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"openshift-service-ca.crt\""
Oct 30 13:24:21.082284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.079695    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"kube-root-ca.crt\""
Oct 30 13:24:21.082284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.079779    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-88e9f3\""
Oct 30 13:24:21.088493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.088449    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"]
Oct 30 13:24:21.174736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.160481    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"agc-cred-e72bf05f28f3\""
Oct 30 13:24:21.174736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.160672    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"payload-processing-external-model-e2e--69d1522b-dockercfg-kbfsv\""
Oct 30 13:24:21.275292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.272600    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 13:24:21.334246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.288116    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.334246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.288216    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.334246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.288331    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strmj\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-kube-api-access-strmj\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.334246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.288377    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.390261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.389831    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-credential-socket\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.389915    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-token\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-token\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.389966    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-certs\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390020    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-data\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390154    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-envoy\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390263    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-socket\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390311    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-podinfo\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390393    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istiod-ca-cert\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390423    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-kube-api-access-t4b2n\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-kube-api-access-t4b2n\") pod \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\" (UID: \"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b\") "
Oct 30 13:24:21.390743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390668    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.390949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390747    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.390949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390881    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strmj\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-kube-api-access-strmj\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.390949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.390936    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.392310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.391282    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-certs" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.392310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.391989    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-socket" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.392534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.392422    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.392819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.392794    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-data" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.393965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.393926    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.394137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.394010    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istiod-ca-cert" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:21.394716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.394660    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-credential-socket" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.398032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.398004    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.398157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.398139    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-kube-api-access-t4b2n" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "kube-api-access-t4b2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:21.398258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.398201    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-envoy" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.399151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.399122    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-podinfo" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:21.400584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.400540    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-token" pod "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" (UID: "e197a8f0-a758-4a8f-a88c-bbc09e1ea50b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:21.448013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.447972    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-strmj\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-kube-api-access-strmj\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535595    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535639    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535654    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535668    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535682    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4b2n\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-kube-api-access-t4b2n\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535697    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535710    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535723    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.535840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.535735    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.688467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.688434    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:21.693285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:21.693257    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c482043e-237d-4780-95eb-ee9dedf162bc" path="/var/lib/kubelet/pods/c482043e-237d-4780-95eb-ee9dedf162bc/volumes"
Oct 30 13:24:22.013687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:22.013646    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"]
Oct 30 13:24:22.024932 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:24:22.024904    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e6b2a7_bbdb_486b_84fd_1c0d038482e2.slice/crio-0658e63bdeb35aa4e0313dfeb45f8f17cfadbb29028d3021f0785c7a9b2d107e WatchSource:0}: Error finding container 0658e63bdeb35aa4e0313dfeb45f8f17cfadbb29028d3021f0785c7a9b2d107e: Status 404 returned error can't find the container with id 0658e63bdeb35aa4e0313dfeb45f8f17cfadbb29028d3021f0785c7a9b2d107e
Oct 30 13:24:22.028157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:22.028035    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"
Oct 30 13:24:22.028157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:22.028046    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct" event={"ID":"e197a8f0-a758-4a8f-a88c-bbc09e1ea50b","Type":"ContainerDied","Data":"bcdcbebe4d208a8647a7f43ef3035130f2d8036370d9ce068fad80710af0e183"}
Oct 30 13:24:22.028157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:22.028110    4993 scope.go:122] "RemoveContainer" containerID="174bda2ab6ac29a784a66e8657d23d6b39173c73a8cf3993c1c51c488a2a1ef2"
Oct 30 13:24:22.090668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:22.090632    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"]
Oct 30 13:24:22.097702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:22.097668    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-kt8ct"]
Oct 30 13:24:23.073242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:23.071847    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t" event={"ID":"40e6b2a7-bbdb-486b-84fd-1c0d038482e2","Type":"ContainerStarted","Data":"b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6"}
Oct 30 13:24:23.073242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:23.071975    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t" event={"ID":"40e6b2a7-bbdb-486b-84fd-1c0d038482e2","Type":"ContainerStarted","Data":"0658e63bdeb35aa4e0313dfeb45f8f17cfadbb29028d3021f0785c7a9b2d107e"}
Oct 30 13:24:23.073242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:23.073172    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:23.692878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:23.692844    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" path="/var/lib/kubelet/pods/e197a8f0-a758-4a8f-a88c-bbc09e1ea50b/volumes"
Oct 30 13:24:27.108229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.102376    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 13:24:27.130547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.130496    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:27.144253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.143108    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 13:24:27.290821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.290358    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:27.425668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.418612    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t" podStartSLOduration=6.418589221 podStartE2EDuration="6.418589221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:23.144594833 +0000 UTC m=+3127.620314443" watchObservedRunningTime="2026-10-30 13:24:27.418589221 +0000 UTC m=+3131.894308836"
Oct 30 13:24:27.431308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.431260    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"]
Oct 30 13:24:27.438304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.438268    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy"
Oct 30 13:24:27.438569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.438551    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy"
Oct 30 13:24:27.438907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.438889    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e197a8f0-a758-4a8f-a88c-bbc09e1ea50b" containerName="istio-proxy"
Oct 30 13:24:27.445969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.445942    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.476173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.476136    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"]
Oct 30 13:24:27.479528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.479500    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"agc-cred-0451c78b5606\""
Oct 30 13:24:27.483228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.479800    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"kube-root-ca.crt\""
Oct 30 13:24:27.500037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.499992    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-63499e\""
Oct 30 13:24:27.507234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.500422    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"routing-overlay\""
Oct 30 13:24:27.507651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.507622    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"openshift-service-ca.crt\""
Oct 30 13:24:27.531314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.508771    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"payload-processing-external-model-e2e--8c883ad3-dockercfg-zkb8p\""
Oct 30 13:24:27.612251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.553840    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.612251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.553942    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.612251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.553982    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.612251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.554268    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9tm\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-kube-api-access-mv9tm\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.667267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.655494    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.730795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.664358    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.731229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.730816    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.731538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.731405    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.731926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.731560    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.732079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.731903    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9tm\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-kube-api-access-mv9tm\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.742529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.741512    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:24:27.795791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.764792    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.805286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.804926    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9tm\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-kube-api-access-mv9tm\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.876233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.864476    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:27.905803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.905762    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"]
Oct 30 13:24:27.909099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.909073    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:27.929086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.929049    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"kube-root-ca.crt\""
Oct 30 13:24:27.929371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.929357    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-6e5458768ddf\""
Oct 30 13:24:27.932056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.932021    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-091b9769f7e9\""
Oct 30 13:24:27.976398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:27.949604    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"]
Oct 30 13:24:28.030686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.030479    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-a4617c99fc7d\""
Oct 30 13:24:28.030686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.030542    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"openshift-service-ca.crt\""
Oct 30 13:24:28.030686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.030634    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8740d4\""
Oct 30 13:24:28.030686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.030646    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"routing-overlay\""
Oct 30 13:24:28.031271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.031245    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-e2e--77992003-dockercfg-zp5z8\""
Oct 30 13:24:28.037595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.037567    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.037733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.037663    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.037813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.037778    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.037922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.037879    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pst8\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-kube-api-access-8pst8\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.140191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.140145    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.140705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.140286    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.140705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.140341    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.140705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.140403    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pst8\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-kube-api-access-8pst8\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.140988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.140961    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.141038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.140996    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.143790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.143760    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.227549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.222336    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pst8\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-kube-api-access-8pst8\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.231069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.231027    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:28.286110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.286066    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"]
Oct 30 13:24:28.292187 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:24:28.292148    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a93d22d_3757_4856_8455_b0d2360db686.slice/crio-03af9a9ba46ffec7ec43542966e4305484ffc104bd959932651af13a000ca7bc WatchSource:0}: Error finding container 03af9a9ba46ffec7ec43542966e4305484ffc104bd959932651af13a000ca7bc: Status 404 returned error can't find the container with id 03af9a9ba46ffec7ec43542966e4305484ffc104bd959932651af13a000ca7bc
Oct 30 13:24:28.587986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:28.587467    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"]
Oct 30 13:24:28.590080 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:24:28.590046    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b24bf5_8a04_4844_b4f4_8af42a10a8b6.slice/crio-f87cd157a1f97da512fe4ff08c07f0e906b06136db2ec3031b1a97332c403be2 WatchSource:0}: Error finding container f87cd157a1f97da512fe4ff08c07f0e906b06136db2ec3031b1a97332c403be2: Status 404 returned error can't find the container with id f87cd157a1f97da512fe4ff08c07f0e906b06136db2ec3031b1a97332c403be2
Oct 30 13:24:29.307548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:29.307505    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj" event={"ID":"23b24bf5-8a04-4844-b4f4-8af42a10a8b6","Type":"ContainerStarted","Data":"f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12"}
Oct 30 13:24:29.308083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:29.307574    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj" event={"ID":"23b24bf5-8a04-4844-b4f4-8af42a10a8b6","Type":"ContainerStarted","Data":"f87cd157a1f97da512fe4ff08c07f0e906b06136db2ec3031b1a97332c403be2"}
Oct 30 13:24:29.308083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:29.307622    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:29.309565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:29.309532    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm" event={"ID":"3a93d22d-3757-4856-8455-b0d2360db686","Type":"ContainerStarted","Data":"870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957"}
Oct 30 13:24:29.309700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:29.309582    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm" event={"ID":"3a93d22d-3757-4856-8455-b0d2360db686","Type":"ContainerStarted","Data":"03af9a9ba46ffec7ec43542966e4305484ffc104bd959932651af13a000ca7bc"}
Oct 30 13:24:29.309700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:29.309603    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:29.357238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:29.357116    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj" podStartSLOduration=2.357093534 podStartE2EDuration="2.357093534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:29.354710782 +0000 UTC m=+3133.830430421" watchObservedRunningTime="2026-10-30 13:24:29.357093534 +0000 UTC m=+3133.832813145"
Oct 30 13:24:29.404231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:29.401463    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm" podStartSLOduration=2.40144229 podStartE2EDuration="2.40144229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:29.400741173 +0000 UTC m=+3133.876460784" watchObservedRunningTime="2026-10-30 13:24:29.40144229 +0000 UTC m=+3133.877161900"
Oct 30 13:24:34.656882 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:34.656778    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"]
Oct 30 13:24:34.661236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:34.661193    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:35.133529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:35.133496    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:24:35.416133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:35.416031    4993 generic.go:356] "Generic (PLEG): container finished" podID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerID="1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd" exitCode=0
Oct 30 13:24:35.416133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:35.416104    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" event={"ID":"7b5ff6f0-dce9-4419-aa9e-abe3045212f7","Type":"ContainerDied","Data":"1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd"}
Oct 30 13:24:35.416650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:35.416631    4993 scope.go:122] "RemoveContainer" containerID="1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd"
Oct 30 13:24:35.418660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:35.418631    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:24:35.418760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:35.418702    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:24:35.418760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:35.418738    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:24:36.438233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:36.435648    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" event={"ID":"7b5ff6f0-dce9-4419-aa9e-abe3045212f7","Type":"ContainerStarted","Data":"8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf"}
Oct 30 13:24:37.185483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:37.185387    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:24:37.186914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:37.186882    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:24:37.187180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:37.187154    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 13:24:38.131010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:38.130973    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:24:38.183677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:38.183620    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:24:38.183876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:38.183718    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 13:24:38.603082 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:24:38.603040    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:24:39.182720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:39.182677    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:24:39.183386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:39.183340    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 13:24:39.658347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:39.658288    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm" podUID="3a93d22d-3757-4856-8455-b0d2360db686" containerName="payload-processing" containerID="cri-o://870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957" gracePeriod=25
Oct 30 13:24:40.107097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.107062    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:40.151111 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.151066    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-plugins-config-volume\") pod \"3a93d22d-3757-4856-8455-b0d2360db686\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") "
Oct 30 13:24:40.151482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.151446    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-provider-credentials\") pod \"3a93d22d-3757-4856-8455-b0d2360db686\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") "
Oct 30 13:24:40.151604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.151491    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-kube-api-access-mv9tm\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-kube-api-access-mv9tm\") pod \"3a93d22d-3757-4856-8455-b0d2360db686\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") "
Oct 30 13:24:40.151604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.151578    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-routing-overlay-volume\") pod \"3a93d22d-3757-4856-8455-b0d2360db686\" (UID: \"3a93d22d-3757-4856-8455-b0d2360db686\") "
Oct 30 13:24:40.151604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.151580    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-plugins-config-volume" pod "3a93d22d-3757-4856-8455-b0d2360db686" (UID: "3a93d22d-3757-4856-8455-b0d2360db686"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:40.152138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.152111    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:40.181246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.179305    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-routing-overlay-volume" pod "3a93d22d-3757-4856-8455-b0d2360db686" (UID: "3a93d22d-3757-4856-8455-b0d2360db686"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:40.210309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.209451    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-provider-credentials" pod "3a93d22d-3757-4856-8455-b0d2360db686" (UID: "3a93d22d-3757-4856-8455-b0d2360db686"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:40.210309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.209531    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-kube-api-access-mv9tm" pod "3a93d22d-3757-4856-8455-b0d2360db686" (UID: "3a93d22d-3757-4856-8455-b0d2360db686"). InnerVolumeSpecName "kube-api-access-mv9tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:40.210309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.209593    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:24:40.210309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.209632    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 13:24:40.283249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.281008    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:40.283249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.281062    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv9tm\" (UniqueName: \"kubernetes.io/projected/3a93d22d-3757-4856-8455-b0d2360db686-kube-api-access-mv9tm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:40.283249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.281080    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a93d22d-3757-4856-8455-b0d2360db686-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:40.387652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.387619    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:24:40.514725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.514690    4993 generic.go:356] "Generic (PLEG): container finished" podID="3a93d22d-3757-4856-8455-b0d2360db686" containerID="870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957" exitCode=0
Oct 30 13:24:40.515165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.515137    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm" event={"ID":"3a93d22d-3757-4856-8455-b0d2360db686","Type":"ContainerDied","Data":"870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957"}
Oct 30 13:24:40.515356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.515291    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"
Oct 30 13:24:40.515448 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.515342    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm" event={"ID":"3a93d22d-3757-4856-8455-b0d2360db686","Type":"ContainerDied","Data":"03af9a9ba46ffec7ec43542966e4305484ffc104bd959932651af13a000ca7bc"}
Oct 30 13:24:40.515550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.515533    4993 scope.go:122] "RemoveContainer" containerID="870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957"
Oct 30 13:24:40.532528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.532469    4993 scope.go:122] "RemoveContainer" containerID="870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957"
Oct 30 13:24:40.532947 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:24:40.532922    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957\": container with ID starting with 870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957 not found: ID does not exist" containerID="870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957"
Oct 30 13:24:40.533041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.532962    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957"} err="failed to get container status \"870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957\": rpc error: code = NotFound desc = could not find container \"870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957\": container with ID starting with 870188cfaee8252c1f9393ef0d5532444f435f17a7708566997a2658b300f957 not found: ID does not exist"
Oct 30 13:24:40.595820 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.595782    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"]
Oct 30 13:24:40.599949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:40.599898    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-f568tq8sm"]
Oct 30 13:24:41.190293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.179420    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"]
Oct 30 13:24:41.190293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.182641    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:24:41.190293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.182712    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 13:24:41.190293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.188399    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3a93d22d-3757-4856-8455-b0d2360db686" containerName="payload-processing"
Oct 30 13:24:41.190293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.188437    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a93d22d-3757-4856-8455-b0d2360db686" containerName="payload-processing"
Oct 30 13:24:41.190293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.188729    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a93d22d-3757-4856-8455-b0d2360db686" containerName="payload-processing"
Oct 30 13:24:41.193031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.192544    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.195985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.195854    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"]
Oct 30 13:24:41.208353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.208319    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:24:41.208517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.208391    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:24:41.208517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.208456    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:24:41.208639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.208548    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:24:41.320013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.319970    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.320013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.320021    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6fr\" (UniqueName: \"kubernetes.io/projected/44b1835a-0147-4811-b67d-e24f005665e4-kube-api-access-2z6fr\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.320614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.320340    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.422112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.422058    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6fr\" (UniqueName: \"kubernetes.io/projected/44b1835a-0147-4811-b67d-e24f005665e4-kube-api-access-2z6fr\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.422356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.422318    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.422476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.422457    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.423028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.423003    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.425035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.425010    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.446224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.446124    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6fr\" (UniqueName: \"kubernetes.io/projected/44b1835a-0147-4811-b67d-e24f005665e4-kube-api-access-2z6fr\") pod \"maas-api-revoke-sub-c13d7e35-mwddm\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.522657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.522622    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:41.710080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.709934    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a93d22d-3757-4856-8455-b0d2360db686" path="/var/lib/kubelet/pods/3a93d22d-3757-4856-8455-b0d2360db686/volumes"
Oct 30 13:24:41.890364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:41.890328    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"]
Oct 30 13:24:41.891534 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:24:41.891487    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b1835a_0147_4811_b67d_e24f005665e4.slice/crio-525c8a61f1fc75ac6eb529efed00746dbf25fe3f677798e4136001c5d3113cbd WatchSource:0}: Error finding container 525c8a61f1fc75ac6eb529efed00746dbf25fe3f677798e4136001c5d3113cbd: Status 404 returned error can't find the container with id 525c8a61f1fc75ac6eb529efed00746dbf25fe3f677798e4136001c5d3113cbd
Oct 30 13:24:42.181039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:42.180997    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:24:42.181263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:42.181066    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 13:24:42.541008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:42.540964    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm" event={"ID":"44b1835a-0147-4811-b67d-e24f005665e4","Type":"ContainerStarted","Data":"9985cf87415434cd886e248a0976d027209c768d4be7cf299812d79db2c01fa7"}
Oct 30 13:24:42.541463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:42.541037    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm" event={"ID":"44b1835a-0147-4811-b67d-e24f005665e4","Type":"ContainerStarted","Data":"525c8a61f1fc75ac6eb529efed00746dbf25fe3f677798e4136001c5d3113cbd"}
Oct 30 13:24:42.584229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:42.584131    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm" podStartSLOduration=1.5841109740000001 podStartE2EDuration="1.584110974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:42.581139288 +0000 UTC m=+3147.056858899" watchObservedRunningTime="2026-10-30 13:24:42.584110974 +0000 UTC m=+3147.059830585"
Oct 30 13:24:43.212289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:43.210595    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:24:43.220078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:43.213455    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:24:43.549419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:43.549381    4993 generic.go:356] "Generic (PLEG): container finished" podID="44b1835a-0147-4811-b67d-e24f005665e4" containerID="9985cf87415434cd886e248a0976d027209c768d4be7cf299812d79db2c01fa7" exitCode=0
Oct 30 13:24:43.549940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:43.549469    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm" event={"ID":"44b1835a-0147-4811-b67d-e24f005665e4","Type":"ContainerDied","Data":"9985cf87415434cd886e248a0976d027209c768d4be7cf299812d79db2c01fa7"}
Oct 30 13:24:45.064230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.064090    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:45.170398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.170361    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b1835a-0147-4811-b67d-e24f005665e4-kube-api-access-2z6fr\" (UniqueName: \"kubernetes.io/projected/44b1835a-0147-4811-b67d-e24f005665e4-kube-api-access-2z6fr\") pod \"44b1835a-0147-4811-b67d-e24f005665e4\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") "
Oct 30 13:24:45.170602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.170520    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-xks-ca\") pod \"44b1835a-0147-4811-b67d-e24f005665e4\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") "
Oct 30 13:24:45.170774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.170746    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-openshift-ca\") pod \"44b1835a-0147-4811-b67d-e24f005665e4\" (UID: \"44b1835a-0147-4811-b67d-e24f005665e4\") "
Oct 30 13:24:45.171112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.171083    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-openshift-ca" pod "44b1835a-0147-4811-b67d-e24f005665e4" (UID: "44b1835a-0147-4811-b67d-e24f005665e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:45.171718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.171687    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.172838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.172813    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-xks-ca" pod "44b1835a-0147-4811-b67d-e24f005665e4" (UID: "44b1835a-0147-4811-b67d-e24f005665e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:45.172989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.172970    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b1835a-0147-4811-b67d-e24f005665e4-kube-api-access-2z6fr" pod "44b1835a-0147-4811-b67d-e24f005665e4" (UID: "44b1835a-0147-4811-b67d-e24f005665e4"). InnerVolumeSpecName "kube-api-access-2z6fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:45.273425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.273388    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6fr\" (UniqueName: \"kubernetes.io/projected/44b1835a-0147-4811-b67d-e24f005665e4-kube-api-access-2z6fr\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.273425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.273424    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1835a-0147-4811-b67d-e24f005665e4-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.569064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.568980    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"
Oct 30 13:24:45.569064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.568992    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm" event={"ID":"44b1835a-0147-4811-b67d-e24f005665e4","Type":"ContainerDied","Data":"525c8a61f1fc75ac6eb529efed00746dbf25fe3f677798e4136001c5d3113cbd"}
Oct 30 13:24:45.569303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:45.569076    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="525c8a61f1fc75ac6eb529efed00746dbf25fe3f677798e4136001c5d3113cbd"
Oct 30 13:24:46.415789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.415749    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"]
Oct 30 13:24:46.416362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.416341    4993 cpu_manager.go:414] "containerMap: removing container" podUID="44b1835a-0147-4811-b67d-e24f005665e4" containerName="revoke-keys"
Oct 30 13:24:46.416362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.416357    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b1835a-0147-4811-b67d-e24f005665e4" containerName="revoke-keys"
Oct 30 13:24:46.416481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.416468    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b1835a-0147-4811-b67d-e24f005665e4" containerName="revoke-keys"
Oct 30 13:24:46.419014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.418989    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.433910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.433887    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"routing-overlay\""
Oct 30 13:24:46.434043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.433931    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"payload-processing-external-model-e2e--65402e9a-dockercfg-s9x98\""
Oct 30 13:24:46.446010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.445973    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"kube-root-ca.crt\""
Oct 30 13:24:46.446190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.446080    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0d9115\""
Oct 30 13:24:46.446190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.446138    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"openshift-service-ca.crt\""
Oct 30 13:24:46.451541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.451514    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"]
Oct 30 13:24:46.525231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.499732    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.525231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.499790    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.525231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.499890    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.525231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.499975    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzw6\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-kube-api-access-flzw6\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.558240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.555670    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"agc-cred-5279a7bb75e1\""
Oct 30 13:24:46.632228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.631063    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzw6\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-kube-api-access-flzw6\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.632228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.631232    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.632228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.631276    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.632228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.631405    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.632618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.632448    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.649228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.648804    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.652433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.650633    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.692321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.677230    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzw6\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-kube-api-access-flzw6\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:46.731555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:46.731496    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:47.040811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:47.040775    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"]
Oct 30 13:24:47.045161 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:24:47.045125    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7e605b_f4c7_4004_805e_2e226f5002c5.slice/crio-5df71b3b0f007e9ef093e87ab0d33b9cebcd94eb23459d002b58e0b8191e813d WatchSource:0}: Error finding container 5df71b3b0f007e9ef093e87ab0d33b9cebcd94eb23459d002b58e0b8191e813d: Status 404 returned error can't find the container with id 5df71b3b0f007e9ef093e87ab0d33b9cebcd94eb23459d002b58e0b8191e813d
Oct 30 13:24:47.645790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:47.645743    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs" event={"ID":"5d7e605b-f4c7-4004-805e-2e226f5002c5","Type":"ContainerStarted","Data":"622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7"}
Oct 30 13:24:47.646400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:47.645843    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:24:47.646400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:47.645894    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs" event={"ID":"5d7e605b-f4c7-4004-805e-2e226f5002c5","Type":"ContainerStarted","Data":"5df71b3b0f007e9ef093e87ab0d33b9cebcd94eb23459d002b58e0b8191e813d"}
Oct 30 13:24:47.710010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:47.709497    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs" podStartSLOduration=1.709452122 podStartE2EDuration="1.709452122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:47.699461105 +0000 UTC m=+3152.175180717" watchObservedRunningTime="2026-10-30 13:24:47.709452122 +0000 UTC m=+3152.185171743"
Oct 30 13:24:58.664162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:24:58.664130    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:25:08.678280 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:25:08.675226    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:25:20.123987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:25:20.123946    4993 scope.go:122] "RemoveContainer" containerID="984659af2954b214b7be0cc1ea453f1cf5c7082d6318c0af023778e56a1a720f"
Oct 30 13:25:20.136656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:25:20.136630    4993 scope.go:122] "RemoveContainer" containerID="d086bf9effa59850366f7ff5931dab3148dc3c416ac06152cb826d8b6c457672"
Oct 30 13:25:20.146453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:25:20.146426    4993 scope.go:122] "RemoveContainer" containerID="ba7bc8055d105ffebeacbca7a4dbc7afa0b388806782da5a38fe7c9a43053d03"
Oct 30 13:25:20.157011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:25:20.156991    4993 scope.go:122] "RemoveContainer" containerID="358ab6e6c4e07bb3393368a1cf2346d5def2b8e6b2eb21e23846ac5427f54530"
Oct 30 13:25:38.569002 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:25:38.568951    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:26:08.569958 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:26:08.569920    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:26:10.631745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.631710    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"]
Oct 30 13:26:10.635903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.635881    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.658030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.657999    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:26:10.663864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.663838    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"]
Oct 30 13:26:10.669852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.669831    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:26:10.670145 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.670121    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:26:10.670302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.670171    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:26:10.788543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.788502    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52lhg\" (UniqueName: \"kubernetes.io/projected/d0e3d750-c6a6-4f66-8926-877b792c0f9e-kube-api-access-52lhg\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.788909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.788564    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.788991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.788917    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.890642 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.890550    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.890807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.890716    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52lhg\" (UniqueName: \"kubernetes.io/projected/d0e3d750-c6a6-4f66-8926-877b792c0f9e-kube-api-access-52lhg\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.890807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.890752    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.891393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.891365    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.893365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.893341    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.913626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.913599    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lhg\" (UniqueName: \"kubernetes.io/projected/d0e3d750-c6a6-4f66-8926-877b792c0f9e-kube-api-access-52lhg\") pod \"maas-api-revoke-sub-d4af9f1c-p2vks\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:10.955249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:10.951462    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:11.279791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:11.279754    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"]
Oct 30 13:26:11.286368 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:26:11.286322    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e3d750_c6a6_4f66_8926_877b792c0f9e.slice/crio-8a1ed57b5bce04e13eb37f233df6a2837d4d6eb7ae5b2432ba0f479d02bdd75c WatchSource:0}: Error finding container 8a1ed57b5bce04e13eb37f233df6a2837d4d6eb7ae5b2432ba0f479d02bdd75c: Status 404 returned error can't find the container with id 8a1ed57b5bce04e13eb37f233df6a2837d4d6eb7ae5b2432ba0f479d02bdd75c
Oct 30 13:26:11.510432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:11.510397    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks" event={"ID":"d0e3d750-c6a6-4f66-8926-877b792c0f9e","Type":"ContainerStarted","Data":"8a1ed57b5bce04e13eb37f233df6a2837d4d6eb7ae5b2432ba0f479d02bdd75c"}
Oct 30 13:26:12.519148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:12.519114    4993 generic.go:356] "Generic (PLEG): container finished" podID="d0e3d750-c6a6-4f66-8926-877b792c0f9e" containerID="b14e9e71a6bb4b9a7f576b3bdd66b7e335c299af5f4fa986331c40aaac8f47ec" exitCode=0
Oct 30 13:26:12.519627 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:12.519158    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks" event={"ID":"d0e3d750-c6a6-4f66-8926-877b792c0f9e","Type":"ContainerDied","Data":"b14e9e71a6bb4b9a7f576b3bdd66b7e335c299af5f4fa986331c40aaac8f47ec"}
Oct 30 13:26:13.686493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.686461    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:13.829266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.829175    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-xks-ca\") pod \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") "
Oct 30 13:26:13.829477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.829304    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e3d750-c6a6-4f66-8926-877b792c0f9e-kube-api-access-52lhg\" (UniqueName: \"kubernetes.io/projected/d0e3d750-c6a6-4f66-8926-877b792c0f9e-kube-api-access-52lhg\") pod \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") "
Oct 30 13:26:13.829477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.829337    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-openshift-ca\") pod \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\" (UID: \"d0e3d750-c6a6-4f66-8926-877b792c0f9e\") "
Oct 30 13:26:13.829837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.829801    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-openshift-ca" pod "d0e3d750-c6a6-4f66-8926-877b792c0f9e" (UID: "d0e3d750-c6a6-4f66-8926-877b792c0f9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:13.829978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.829896    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:13.831835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.831812    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-xks-ca" pod "d0e3d750-c6a6-4f66-8926-877b792c0f9e" (UID: "d0e3d750-c6a6-4f66-8926-877b792c0f9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:13.831835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.831825    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e3d750-c6a6-4f66-8926-877b792c0f9e-kube-api-access-52lhg" pod "d0e3d750-c6a6-4f66-8926-877b792c0f9e" (UID: "d0e3d750-c6a6-4f66-8926-877b792c0f9e"). InnerVolumeSpecName "kube-api-access-52lhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:13.930833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.930800    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e3d750-c6a6-4f66-8926-877b792c0f9e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:13.930833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:13.930833    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52lhg\" (UniqueName: \"kubernetes.io/projected/d0e3d750-c6a6-4f66-8926-877b792c0f9e-kube-api-access-52lhg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.535115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:14.535074    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks" event={"ID":"d0e3d750-c6a6-4f66-8926-877b792c0f9e","Type":"ContainerDied","Data":"8a1ed57b5bce04e13eb37f233df6a2837d4d6eb7ae5b2432ba0f479d02bdd75c"}
Oct 30 13:26:14.535115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:14.535098    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"
Oct 30 13:26:14.535390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:14.535133    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a1ed57b5bce04e13eb37f233df6a2837d4d6eb7ae5b2432ba0f479d02bdd75c"
Oct 30 13:26:17.702538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:17.702500    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"]
Oct 30 13:26:20.211893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:20.211856    4993 scope.go:122] "RemoveContainer" containerID="f640e096627611006477e12850a7fa4acc0280854b7dba470d9a6de7b71e7871"
Oct 30 13:26:22.300522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:22.300468    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"]
Oct 30 13:26:22.723314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:22.723169    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj" podUID="23b24bf5-8a04-4844-b4f4-8af42a10a8b6" containerName="payload-processing" containerID="cri-o://f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12" gracePeriod=25
Oct 30 13:26:23.083994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.083966    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:26:23.134405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.134370    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-plugins-config-volume\") pod \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") "
Oct 30 13:26:23.134611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.134592    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-kube-api-access-8pst8\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-kube-api-access-8pst8\") pod \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") "
Oct 30 13:26:23.134676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.134628    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-provider-credentials\") pod \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") "
Oct 30 13:26:23.134676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.134671    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-routing-overlay-volume\") pod \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\" (UID: \"23b24bf5-8a04-4844-b4f4-8af42a10a8b6\") "
Oct 30 13:26:23.135145 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.135116    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-plugins-config-volume" pod "23b24bf5-8a04-4844-b4f4-8af42a10a8b6" (UID: "23b24bf5-8a04-4844-b4f4-8af42a10a8b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:23.135321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.135303    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:23.135562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.135539    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-routing-overlay-volume" pod "23b24bf5-8a04-4844-b4f4-8af42a10a8b6" (UID: "23b24bf5-8a04-4844-b4f4-8af42a10a8b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:23.144307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.144272    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-kube-api-access-8pst8" pod "23b24bf5-8a04-4844-b4f4-8af42a10a8b6" (UID: "23b24bf5-8a04-4844-b4f4-8af42a10a8b6"). InnerVolumeSpecName "kube-api-access-8pst8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:23.177091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.164421    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-provider-credentials" pod "23b24bf5-8a04-4844-b4f4-8af42a10a8b6" (UID: "23b24bf5-8a04-4844-b4f4-8af42a10a8b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:23.244605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.244573    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pst8\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-kube-api-access-8pst8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:23.244605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.244605    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:23.244846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.244614    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b24bf5-8a04-4844-b4f4-8af42a10a8b6-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:23.692700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.692664    4993 generic.go:356] "Generic (PLEG): container finished" podID="23b24bf5-8a04-4844-b4f4-8af42a10a8b6" containerID="f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12" exitCode=0
Oct 30 13:26:23.693256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.692794    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"
Oct 30 13:26:23.694301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.694273    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj" event={"ID":"23b24bf5-8a04-4844-b4f4-8af42a10a8b6","Type":"ContainerDied","Data":"f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12"}
Oct 30 13:26:23.694443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.694322    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj" event={"ID":"23b24bf5-8a04-4844-b4f4-8af42a10a8b6","Type":"ContainerDied","Data":"f87cd157a1f97da512fe4ff08c07f0e906b06136db2ec3031b1a97332c403be2"}
Oct 30 13:26:23.694443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.694354    4993 scope.go:122] "RemoveContainer" containerID="f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12"
Oct 30 13:26:23.706504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.706486    4993 scope.go:122] "RemoveContainer" containerID="f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12"
Oct 30 13:26:23.706953 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:26:23.706929    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12\": container with ID starting with f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12 not found: ID does not exist" containerID="f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12"
Oct 30 13:26:23.707033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.706962    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12"} err="failed to get container status \"f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12\": rpc error: code = NotFound desc = could not find container \"f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12\": container with ID starting with f5014a2c6d11f99a91681a54e288d81290dab246a1ec8905388d74523f005c12 not found: ID does not exist"
Oct 30 13:26:23.745245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.745194    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"]
Oct 30 13:26:23.748960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:23.748934    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595ddtsmj"]
Oct 30 13:26:25.693021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:25.692987    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b24bf5-8a04-4844-b4f4-8af42a10a8b6" path="/var/lib/kubelet/pods/23b24bf5-8a04-4844-b4f4-8af42a10a8b6/volumes"
Oct 30 13:26:26.737180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.737143    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"]
Oct 30 13:26:26.737763 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.737747    4993 cpu_manager.go:414] "containerMap: removing container" podUID="23b24bf5-8a04-4844-b4f4-8af42a10a8b6" containerName="payload-processing"
Oct 30 13:26:26.737805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.737764    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b24bf5-8a04-4844-b4f4-8af42a10a8b6" containerName="payload-processing"
Oct 30 13:26:26.737805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.737789    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d0e3d750-c6a6-4f66-8926-877b792c0f9e" containerName="revoke-keys"
Oct 30 13:26:26.737805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.737795    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e3d750-c6a6-4f66-8926-877b792c0f9e" containerName="revoke-keys"
Oct 30 13:26:26.737907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.737880    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e3d750-c6a6-4f66-8926-877b792c0f9e" containerName="revoke-keys"
Oct 30 13:26:26.737907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.737896    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b24bf5-8a04-4844-b4f4-8af42a10a8b6" containerName="payload-processing"
Oct 30 13:26:26.742960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.742937    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.744247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.744189    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.744375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.744317    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vhd\" (UniqueName: \"kubernetes.io/projected/17e52616-6f4a-4c50-833c-dd3e4ebd4208-kube-api-access-88vhd\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.744477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.744451    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.761868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.761833    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"]
Oct 30 13:26:26.768178 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.768148    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:26:26.768426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.768161    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:26:26.768426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.768314    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:26:26.769201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.769179    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:26:26.845780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.845742    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.846140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.845802    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vhd\" (UniqueName: \"kubernetes.io/projected/17e52616-6f4a-4c50-833c-dd3e4ebd4208-kube-api-access-88vhd\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.846140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.845855    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.846881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.846557    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.848487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.848457    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:26.869751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:26.869711    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vhd\" (UniqueName: \"kubernetes.io/projected/17e52616-6f4a-4c50-833c-dd3e4ebd4208-kube-api-access-88vhd\") pod \"maas-api-revoke-sub-9bf29d12-k5hzs\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:27.057850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.057750    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:27.302474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.302426    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t" podUID="40e6b2a7-bbdb-486b-84fd-1c0d038482e2" containerName="payload-processing" containerID="cri-o://b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6" gracePeriod=25
Oct 30 13:26:27.369493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.369460    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"]
Oct 30 13:26:27.393496 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:26:27.393455    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e52616_6f4a_4c50_833c_dd3e4ebd4208.slice/crio-4e7cd39800a88b8b35f4003169201332bd0dde597baf1580fd3e9b9fb33d1ffd WatchSource:0}: Error finding container 4e7cd39800a88b8b35f4003169201332bd0dde597baf1580fd3e9b9fb33d1ffd: Status 404 returned error can't find the container with id 4e7cd39800a88b8b35f4003169201332bd0dde597baf1580fd3e9b9fb33d1ffd
Oct 30 13:26:27.564002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.563966    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:26:27.666444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.666415    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-provider-credentials\") pod \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") "
Oct 30 13:26:27.666687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.666668    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-plugins-config-volume\") pod \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") "
Oct 30 13:26:27.666835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.666817    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-routing-overlay-volume\") pod \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") "
Oct 30 13:26:27.667666 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.667327    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-plugins-config-volume" pod "40e6b2a7-bbdb-486b-84fd-1c0d038482e2" (UID: "40e6b2a7-bbdb-486b-84fd-1c0d038482e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:27.667760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.667577    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-routing-overlay-volume" pod "40e6b2a7-bbdb-486b-84fd-1c0d038482e2" (UID: "40e6b2a7-bbdb-486b-84fd-1c0d038482e2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:27.670558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.669710    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-kube-api-access-strmj\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-kube-api-access-strmj\") pod \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\" (UID: \"40e6b2a7-bbdb-486b-84fd-1c0d038482e2\") "
Oct 30 13:26:27.670558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.670511    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:27.670558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.670534    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:27.671076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.670930    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-provider-credentials" pod "40e6b2a7-bbdb-486b-84fd-1c0d038482e2" (UID: "40e6b2a7-bbdb-486b-84fd-1c0d038482e2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:27.672920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.672897    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-kube-api-access-strmj" pod "40e6b2a7-bbdb-486b-84fd-1c0d038482e2" (UID: "40e6b2a7-bbdb-486b-84fd-1c0d038482e2"). InnerVolumeSpecName "kube-api-access-strmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:27.727805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.727764    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs" event={"ID":"17e52616-6f4a-4c50-833c-dd3e4ebd4208","Type":"ContainerStarted","Data":"bedd1f9ac081b5cc26e397f36b9d1f3a467ea305721e507f061aa42d9827e084"}
Oct 30 13:26:27.728014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.727827    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs" event={"ID":"17e52616-6f4a-4c50-833c-dd3e4ebd4208","Type":"ContainerStarted","Data":"4e7cd39800a88b8b35f4003169201332bd0dde597baf1580fd3e9b9fb33d1ffd"}
Oct 30 13:26:27.729651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.729625    4993 generic.go:356] "Generic (PLEG): container finished" podID="40e6b2a7-bbdb-486b-84fd-1c0d038482e2" containerID="b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6" exitCode=0
Oct 30 13:26:27.729812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.729686    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"
Oct 30 13:26:27.729812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.729721    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t" event={"ID":"40e6b2a7-bbdb-486b-84fd-1c0d038482e2","Type":"ContainerDied","Data":"b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6"}
Oct 30 13:26:27.729812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.729778    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t" event={"ID":"40e6b2a7-bbdb-486b-84fd-1c0d038482e2","Type":"ContainerDied","Data":"0658e63bdeb35aa4e0313dfeb45f8f17cfadbb29028d3021f0785c7a9b2d107e"}
Oct 30 13:26:27.729812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.729807    4993 scope.go:122] "RemoveContainer" containerID="b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6"
Oct 30 13:26:27.740189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.740171    4993 scope.go:122] "RemoveContainer" containerID="b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6"
Oct 30 13:26:27.740524 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:26:27.740497    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6\": container with ID starting with b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6 not found: ID does not exist" containerID="b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6"
Oct 30 13:26:27.740569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.740533    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6"} err="failed to get container status \"b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6\": rpc error: code = NotFound desc = could not find container \"b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6\": container with ID starting with b9fbfe8a77d4f128076a0b8d7d080ef1e08ba12c808748c4f343fcaf7ade04a6 not found: ID does not exist"
Oct 30 13:26:27.771922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.771887    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-strmj\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-kube-api-access-strmj\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:27.772153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.772135    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e6b2a7-bbdb-486b-84fd-1c0d038482e2-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:27.772866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.772816    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs" podStartSLOduration=1.7728040969999999 podStartE2EDuration="1.772804097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:27.769830143 +0000 UTC m=+3252.245549753" watchObservedRunningTime="2026-10-30 13:26:27.772804097 +0000 UTC m=+3252.248523706"
Oct 30 13:26:27.808287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.808252    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"]
Oct 30 13:26:27.813796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:27.813767    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-ccbf5wq9t"]
Oct 30 13:26:28.738595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:28.738561    4993 generic.go:356] "Generic (PLEG): container finished" podID="17e52616-6f4a-4c50-833c-dd3e4ebd4208" containerID="bedd1f9ac081b5cc26e397f36b9d1f3a467ea305721e507f061aa42d9827e084" exitCode=0
Oct 30 13:26:28.738764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:28.738647    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs" event={"ID":"17e52616-6f4a-4c50-833c-dd3e4ebd4208","Type":"ContainerDied","Data":"bedd1f9ac081b5cc26e397f36b9d1f3a467ea305721e507f061aa42d9827e084"}
Oct 30 13:26:29.692263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:29.692229    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e6b2a7-bbdb-486b-84fd-1c0d038482e2" path="/var/lib/kubelet/pods/40e6b2a7-bbdb-486b-84fd-1c0d038482e2/volumes"
Oct 30 13:26:29.940322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:29.940285    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:30.006535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.006496    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-openshift-ca\") pod \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") "
Oct 30 13:26:30.006734 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.006704    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e52616-6f4a-4c50-833c-dd3e4ebd4208-kube-api-access-88vhd\" (UniqueName: \"kubernetes.io/projected/17e52616-6f4a-4c50-833c-dd3e4ebd4208-kube-api-access-88vhd\") pod \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") "
Oct 30 13:26:30.006792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.006745    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-xks-ca\") pod \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\" (UID: \"17e52616-6f4a-4c50-833c-dd3e4ebd4208\") "
Oct 30 13:26:30.006956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.006928    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-openshift-ca" pod "17e52616-6f4a-4c50-833c-dd3e4ebd4208" (UID: "17e52616-6f4a-4c50-833c-dd3e4ebd4208"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:30.007476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.007446    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:30.011121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.010098    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-xks-ca" pod "17e52616-6f4a-4c50-833c-dd3e4ebd4208" (UID: "17e52616-6f4a-4c50-833c-dd3e4ebd4208"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:30.016630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.016596    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e52616-6f4a-4c50-833c-dd3e4ebd4208-kube-api-access-88vhd" pod "17e52616-6f4a-4c50-833c-dd3e4ebd4208" (UID: "17e52616-6f4a-4c50-833c-dd3e4ebd4208"). InnerVolumeSpecName "kube-api-access-88vhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:30.111698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.110286    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88vhd\" (UniqueName: \"kubernetes.io/projected/17e52616-6f4a-4c50-833c-dd3e4ebd4208-kube-api-access-88vhd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:30.111975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.111951    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e52616-6f4a-4c50-833c-dd3e4ebd4208-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:30.755714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.755682    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"
Oct 30 13:26:30.756268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.755722    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs" event={"ID":"17e52616-6f4a-4c50-833c-dd3e4ebd4208","Type":"ContainerDied","Data":"4e7cd39800a88b8b35f4003169201332bd0dde597baf1580fd3e9b9fb33d1ffd"}
Oct 30 13:26:30.756268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:30.755783    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7cd39800a88b8b35f4003169201332bd0dde597baf1580fd3e9b9fb33d1ffd"
Oct 30 13:26:33.982141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:33.982099    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"]
Oct 30 13:26:33.982965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:33.982944    4993 cpu_manager.go:414] "containerMap: removing container" podUID="40e6b2a7-bbdb-486b-84fd-1c0d038482e2" containerName="payload-processing"
Oct 30 13:26:33.983024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:33.982968    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e6b2a7-bbdb-486b-84fd-1c0d038482e2" containerName="payload-processing"
Oct 30 13:26:33.983073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:33.983053    4993 cpu_manager.go:414] "containerMap: removing container" podUID="17e52616-6f4a-4c50-833c-dd3e4ebd4208" containerName="revoke-keys"
Oct 30 13:26:33.983073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:33.983061    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e52616-6f4a-4c50-833c-dd3e4ebd4208" containerName="revoke-keys"
Oct 30 13:26:33.983191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:33.983178    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e6b2a7-bbdb-486b-84fd-1c0d038482e2" containerName="payload-processing"
Oct 30 13:26:33.983273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:33.983222    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e52616-6f4a-4c50-833c-dd3e4ebd4208" containerName="revoke-keys"
Oct 30 13:26:33.987464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:33.987443    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.003681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.003658    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:26:34.007162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.007133    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"]
Oct 30 13:26:34.014249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.013686    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:26:34.015347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.015326    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:26:34.015575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.015554    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:26:34.070851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.070195    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.070851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.070316    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwjc\" (UniqueName: \"kubernetes.io/projected/d3a9adda-d603-436e-b5fd-a3c6523268d0-kube-api-access-ttwjc\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.070851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.070544    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.171537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.171501    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.171707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.171544    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwjc\" (UniqueName: \"kubernetes.io/projected/d3a9adda-d603-436e-b5fd-a3c6523268d0-kube-api-access-ttwjc\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.171707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.171683    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.172404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.172368    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.174273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.174245    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.195109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.195081    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwjc\" (UniqueName: \"kubernetes.io/projected/d3a9adda-d603-436e-b5fd-a3c6523268d0-kube-api-access-ttwjc\") pod \"maas-api-revoke-sub-ae083fe5-6t9r8\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.305083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.304998    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:34.657284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.657256    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"]
Oct 30 13:26:34.660919 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:26:34.660887    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a9adda_d603_436e_b5fd_a3c6523268d0.slice/crio-0ac80f52284b1f5821bf28840b282bdac550ec82b3d0bd6374150306c50b5733 WatchSource:0}: Error finding container 0ac80f52284b1f5821bf28840b282bdac550ec82b3d0bd6374150306c50b5733: Status 404 returned error can't find the container with id 0ac80f52284b1f5821bf28840b282bdac550ec82b3d0bd6374150306c50b5733
Oct 30 13:26:34.788248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:34.788198    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8" event={"ID":"d3a9adda-d603-436e-b5fd-a3c6523268d0","Type":"ContainerStarted","Data":"0ac80f52284b1f5821bf28840b282bdac550ec82b3d0bd6374150306c50b5733"}
Oct 30 13:26:35.797432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:35.797400    4993 generic.go:356] "Generic (PLEG): container finished" podID="d3a9adda-d603-436e-b5fd-a3c6523268d0" containerID="b0dbd338fccbea0bb1fb4481049cdf6e88d5e3a521aff61f397095c9845e02bf" exitCode=0
Oct 30 13:26:35.797914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:35.797485    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8" event={"ID":"d3a9adda-d603-436e-b5fd-a3c6523268d0","Type":"ContainerDied","Data":"b0dbd338fccbea0bb1fb4481049cdf6e88d5e3a521aff61f397095c9845e02bf"}
Oct 30 13:26:36.993050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:36.992947    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:37.122397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.122315    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-openshift-ca\") pod \"d3a9adda-d603-436e-b5fd-a3c6523268d0\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") "
Oct 30 13:26:37.122570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.122552    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a9adda-d603-436e-b5fd-a3c6523268d0-kube-api-access-ttwjc\" (UniqueName: \"kubernetes.io/projected/d3a9adda-d603-436e-b5fd-a3c6523268d0-kube-api-access-ttwjc\") pod \"d3a9adda-d603-436e-b5fd-a3c6523268d0\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") "
Oct 30 13:26:37.122661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.122644    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-xks-ca\") pod \"d3a9adda-d603-436e-b5fd-a3c6523268d0\" (UID: \"d3a9adda-d603-436e-b5fd-a3c6523268d0\") "
Oct 30 13:26:37.122773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.122737    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-openshift-ca" pod "d3a9adda-d603-436e-b5fd-a3c6523268d0" (UID: "d3a9adda-d603-436e-b5fd-a3c6523268d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:37.123075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.123054    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:37.124781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.124759    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-xks-ca" pod "d3a9adda-d603-436e-b5fd-a3c6523268d0" (UID: "d3a9adda-d603-436e-b5fd-a3c6523268d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:37.124985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.124966    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a9adda-d603-436e-b5fd-a3c6523268d0-kube-api-access-ttwjc" pod "d3a9adda-d603-436e-b5fd-a3c6523268d0" (UID: "d3a9adda-d603-436e-b5fd-a3c6523268d0"). InnerVolumeSpecName "kube-api-access-ttwjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:37.224381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.224342    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttwjc\" (UniqueName: \"kubernetes.io/projected/d3a9adda-d603-436e-b5fd-a3c6523268d0-kube-api-access-ttwjc\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:37.224381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.224374    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a9adda-d603-436e-b5fd-a3c6523268d0-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:37.817124 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.817092    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"
Oct 30 13:26:37.817389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.817118    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8" event={"ID":"d3a9adda-d603-436e-b5fd-a3c6523268d0","Type":"ContainerDied","Data":"0ac80f52284b1f5821bf28840b282bdac550ec82b3d0bd6374150306c50b5733"}
Oct 30 13:26:37.817461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:37.817413    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ac80f52284b1f5821bf28840b282bdac550ec82b3d0bd6374150306c50b5733"
Oct 30 13:26:38.110376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:38.110332    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"]
Oct 30 13:26:38.893229 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:26:38.893093    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:26:39.775851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.772253    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"]
Oct 30 13:26:39.776512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.776186    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d3a9adda-d603-436e-b5fd-a3c6523268d0" containerName="revoke-keys"
Oct 30 13:26:39.776512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.776225    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a9adda-d603-436e-b5fd-a3c6523268d0" containerName="revoke-keys"
Oct 30 13:26:39.776512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.776453    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a9adda-d603-436e-b5fd-a3c6523268d0" containerName="revoke-keys"
Oct 30 13:26:39.780273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.780251    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.794384 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.794350    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"]
Oct 30 13:26:39.799650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.799623    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"routing-overlay\""
Oct 30 13:26:39.801037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.801013    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-acf9e2\""
Oct 30 13:26:39.801166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.801077    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"kube-root-ca.crt\""
Oct 30 13:26:39.801166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.801151    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"openshift-service-ca.crt\""
Oct 30 13:26:39.839052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.839017    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"agc-cred-f967f1739ca3\""
Oct 30 13:26:39.839246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.839087    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"payload-processing-external-model-e2e--66ed339f-dockercfg-5sgmj\""
Oct 30 13:26:39.860708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.860677    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czxj\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-kube-api-access-2czxj\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.860908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.860723    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.860908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.860805    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.860908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.860864    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.967373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.967322    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czxj\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-kube-api-access-2czxj\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.967696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.967664    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.967865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.967844    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.968087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.968062    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.970630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.970597    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:39.971286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:39.970928    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:40.005232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:40.003910    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:40.032168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:40.032033    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czxj\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-kube-api-access-2czxj\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:40.102757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:40.102462    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:40.596344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:40.596310    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"]
Oct 30 13:26:40.597003 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:26:40.596973    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7371fc54_7fdf_4217_a7a7_4ea682e31616.slice/crio-0058030b6371076978adbf9500ed42ef891668b3eeaac05234864dadf97029e0 WatchSource:0}: Error finding container 0058030b6371076978adbf9500ed42ef891668b3eeaac05234864dadf97029e0: Status 404 returned error can't find the container with id 0058030b6371076978adbf9500ed42ef891668b3eeaac05234864dadf97029e0
Oct 30 13:26:40.880270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:40.880126    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9" event={"ID":"7371fc54-7fdf-4217-a7a7-4ea682e31616","Type":"ContainerStarted","Data":"791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f"}
Oct 30 13:26:40.880270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:40.880185    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9" event={"ID":"7371fc54-7fdf-4217-a7a7-4ea682e31616","Type":"ContainerStarted","Data":"0058030b6371076978adbf9500ed42ef891668b3eeaac05234864dadf97029e0"}
Oct 30 13:26:40.880270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:40.880244    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:41.111118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.111045    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9" podStartSLOduration=2.111020488 podStartE2EDuration="2.111020488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:40.935401985 +0000 UTC m=+3265.411121597" watchObservedRunningTime="2026-10-30 13:26:41.111020488 +0000 UTC m=+3265.586740099"
Oct 30 13:26:41.114360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.114325    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"]
Oct 30 13:26:41.119697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.119665    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.134804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.134737    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"agc-cred-b572fbd3f0eb\""
Oct 30 13:26:41.142199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.142167    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"]
Oct 30 13:26:41.249074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.249030    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.249074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.249088    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.249392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.249227    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktll\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-kube-api-access-pktll\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.249392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.249266    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.350423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.350384    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktll\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-kube-api-access-pktll\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.350423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.350432    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.350733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.350601    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.350733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.350644    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.351302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.351265    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.351420 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.351339    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.353394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.353367    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.374546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.374505    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktll\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-kube-api-access-pktll\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.438563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.438467    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:41.796329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.796078    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"]
Oct 30 13:26:41.890169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.890129    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm" event={"ID":"7780546f-5188-4cdf-b5dd-dc1156443f13","Type":"ContainerStarted","Data":"e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db"}
Oct 30 13:26:41.890599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.890190    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm" event={"ID":"7780546f-5188-4cdf-b5dd-dc1156443f13","Type":"ContainerStarted","Data":"7cc791edb911ef4164c11c792529164a5354f86f678e10abcf32a2afad8fcbdb"}
Oct 30 13:26:41.931830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:41.931758    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm" podStartSLOduration=0.931738737 podStartE2EDuration="931.738737ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:41.93083687 +0000 UTC m=+3266.406556480" watchObservedRunningTime="2026-10-30 13:26:41.931738737 +0000 UTC m=+3266.407458347"
Oct 30 13:26:42.897988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:42.897953    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:43.111149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.111095    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs" podUID="5d7e605b-f4c7-4004-805e-2e226f5002c5" containerName="payload-processing" containerID="cri-o://622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7" gracePeriod=25
Oct 30 13:26:43.399498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.399465    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:26:43.498279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.497372    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-routing-overlay-volume\") pod \"5d7e605b-f4c7-4004-805e-2e226f5002c5\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") "
Oct 30 13:26:43.498279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.497594    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-provider-credentials\") pod \"5d7e605b-f4c7-4004-805e-2e226f5002c5\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") "
Oct 30 13:26:43.498279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.497679    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-plugins-config-volume\") pod \"5d7e605b-f4c7-4004-805e-2e226f5002c5\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") "
Oct 30 13:26:43.498279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.497763    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-kube-api-access-flzw6\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-kube-api-access-flzw6\") pod \"5d7e605b-f4c7-4004-805e-2e226f5002c5\" (UID: \"5d7e605b-f4c7-4004-805e-2e226f5002c5\") "
Oct 30 13:26:43.498279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.497816    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-routing-overlay-volume" pod "5d7e605b-f4c7-4004-805e-2e226f5002c5" (UID: "5d7e605b-f4c7-4004-805e-2e226f5002c5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:43.498709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.498471    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:43.498709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.498606    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-plugins-config-volume" pod "5d7e605b-f4c7-4004-805e-2e226f5002c5" (UID: "5d7e605b-f4c7-4004-805e-2e226f5002c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:43.506228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.501641    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-kube-api-access-flzw6" pod "5d7e605b-f4c7-4004-805e-2e226f5002c5" (UID: "5d7e605b-f4c7-4004-805e-2e226f5002c5"). InnerVolumeSpecName "kube-api-access-flzw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:43.507412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.507378    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-provider-credentials" pod "5d7e605b-f4c7-4004-805e-2e226f5002c5" (UID: "5d7e605b-f4c7-4004-805e-2e226f5002c5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:43.607228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.606397    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:43.607228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.606442    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7e605b-f4c7-4004-805e-2e226f5002c5-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:43.607228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.606457    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flzw6\" (UniqueName: \"kubernetes.io/projected/5d7e605b-f4c7-4004-805e-2e226f5002c5-kube-api-access-flzw6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:43.906187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.906096    4993 generic.go:356] "Generic (PLEG): container finished" podID="5d7e605b-f4c7-4004-805e-2e226f5002c5" containerID="622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7" exitCode=0
Oct 30 13:26:43.906187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.906138    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs" event={"ID":"5d7e605b-f4c7-4004-805e-2e226f5002c5","Type":"ContainerDied","Data":"622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7"}
Oct 30 13:26:43.906724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.906201    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs" event={"ID":"5d7e605b-f4c7-4004-805e-2e226f5002c5","Type":"ContainerDied","Data":"5df71b3b0f007e9ef093e87ab0d33b9cebcd94eb23459d002b58e0b8191e813d"}
Oct 30 13:26:43.906724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.906149    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"
Oct 30 13:26:43.906724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.906250    4993 scope.go:122] "RemoveContainer" containerID="622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7"
Oct 30 13:26:43.919923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.919899    4993 scope.go:122] "RemoveContainer" containerID="622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7"
Oct 30 13:26:43.920401 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:26:43.920355    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7\": container with ID starting with 622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7 not found: ID does not exist" containerID="622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7"
Oct 30 13:26:43.920538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.920404    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7"} err="failed to get container status \"622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7\": rpc error: code = NotFound desc = could not find container \"622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7\": container with ID starting with 622cba8d1d670df87da255a13792bc07d2c2527aa55b854a19ea8c8b43fdc5a7 not found: ID does not exist"
Oct 30 13:26:43.959352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.959304    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"]
Oct 30 13:26:43.971874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:43.971839    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-776cc7tjs"]
Oct 30 13:26:45.726252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:45.725179    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7e605b-f4c7-4004-805e-2e226f5002c5" path="/var/lib/kubelet/pods/5d7e605b-f4c7-4004-805e-2e226f5002c5/volumes"
Oct 30 13:26:49.431012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.430973    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"]
Oct 30 13:26:49.431585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.431562    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5d7e605b-f4c7-4004-805e-2e226f5002c5" containerName="payload-processing"
Oct 30 13:26:49.431585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.431579    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7e605b-f4c7-4004-805e-2e226f5002c5" containerName="payload-processing"
Oct 30 13:26:49.431735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.431694    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7e605b-f4c7-4004-805e-2e226f5002c5" containerName="payload-processing"
Oct 30 13:26:49.435585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.435562    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.455899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.453647    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4db150\""
Oct 30 13:26:49.455899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.453769    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"routing-overlay\""
Oct 30 13:26:49.459980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.459950    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"]
Oct 30 13:26:49.466422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.466397    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"agc-cred-fd7af1e203d7\""
Oct 30 13:26:49.466574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.466496    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"kube-root-ca.crt\""
Oct 30 13:26:49.466574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.466561    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"openshift-service-ca.crt\""
Oct 30 13:26:49.466716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.466664    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-e2e--1b2e61a1-dockercfg-m72r8\""
Oct 30 13:26:49.534249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.534195    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.534483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.534452    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgn49\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-kube-api-access-fgn49\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.534569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.534556    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.534629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.534599    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.640228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.639343    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgn49\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-kube-api-access-fgn49\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.640228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.639912    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.644316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.641107    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.644316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.641187    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.655228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.654614    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.655228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.655107    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.700094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.699945    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.734577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.734544    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgn49\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-kube-api-access-fgn49\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:49.752502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:49.752460    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:50.079398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:50.079235    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"]
Oct 30 13:26:50.085448 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:26:50.085416    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c8f749_e8b0_437f_9392_f9ee2bebad97.slice/crio-02f8af64b4c271115d6bbba657529b899b4b6688773182e6ce2d987bc11be04b WatchSource:0}: Error finding container 02f8af64b4c271115d6bbba657529b899b4b6688773182e6ce2d987bc11be04b: Status 404 returned error can't find the container with id 02f8af64b4c271115d6bbba657529b899b4b6688773182e6ce2d987bc11be04b
Oct 30 13:26:51.031931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:51.031891    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9" event={"ID":"05c8f749-e8b0-437f-9392-f9ee2bebad97","Type":"ContainerStarted","Data":"e4e73bca748c8537ef731276e2ac4ce0a95aa787d8e1d9305c76d37297ed7e41"}
Oct 30 13:26:51.032442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:51.031944    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9" event={"ID":"05c8f749-e8b0-437f-9392-f9ee2bebad97","Type":"ContainerStarted","Data":"02f8af64b4c271115d6bbba657529b899b4b6688773182e6ce2d987bc11be04b"}
Oct 30 13:26:51.032442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:51.032008    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:26:51.076461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:51.076387    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9" podStartSLOduration=2.076372218 podStartE2EDuration="2.076372218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:51.072869235 +0000 UTC m=+3275.548588846" watchObservedRunningTime="2026-10-30 13:26:51.076372218 +0000 UTC m=+3275.552091827"
Oct 30 13:26:51.894699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:51.894661    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:53.910345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:53.910305    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:26:54.030596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:54.030552    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"]
Oct 30 13:26:59.030930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.030867    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9" podUID="7371fc54-7fdf-4217-a7a7-4ea682e31616" containerName="payload-processing" containerID="cri-o://791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f" gracePeriod=25
Oct 30 13:26:59.342442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.342413    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:26:59.380552 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.380519    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-routing-overlay-volume\") pod \"7371fc54-7fdf-4217-a7a7-4ea682e31616\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") "
Oct 30 13:26:59.380760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.380653    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-kube-api-access-2czxj\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-kube-api-access-2czxj\") pod \"7371fc54-7fdf-4217-a7a7-4ea682e31616\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") "
Oct 30 13:26:59.380760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.380708    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-plugins-config-volume\") pod \"7371fc54-7fdf-4217-a7a7-4ea682e31616\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") "
Oct 30 13:26:59.380760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.380739    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-provider-credentials\") pod \"7371fc54-7fdf-4217-a7a7-4ea682e31616\" (UID: \"7371fc54-7fdf-4217-a7a7-4ea682e31616\") "
Oct 30 13:26:59.380995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.380969    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-routing-overlay-volume" pod "7371fc54-7fdf-4217-a7a7-4ea682e31616" (UID: "7371fc54-7fdf-4217-a7a7-4ea682e31616"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:59.381128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.381106    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-plugins-config-volume" pod "7371fc54-7fdf-4217-a7a7-4ea682e31616" (UID: "7371fc54-7fdf-4217-a7a7-4ea682e31616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:59.381445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.381394    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:59.381445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.381422    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7371fc54-7fdf-4217-a7a7-4ea682e31616-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:59.383225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.383181    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-kube-api-access-2czxj" pod "7371fc54-7fdf-4217-a7a7-4ea682e31616" (UID: "7371fc54-7fdf-4217-a7a7-4ea682e31616"). InnerVolumeSpecName "kube-api-access-2czxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:59.383371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.383348    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-provider-credentials" pod "7371fc54-7fdf-4217-a7a7-4ea682e31616" (UID: "7371fc54-7fdf-4217-a7a7-4ea682e31616"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:59.482905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.482865    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2czxj\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-kube-api-access-2czxj\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:26:59.482905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:26:59.482906    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7371fc54-7fdf-4217-a7a7-4ea682e31616-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.148879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.148842    4993 generic.go:356] "Generic (PLEG): container finished" podID="7371fc54-7fdf-4217-a7a7-4ea682e31616" containerID="791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f" exitCode=0
Oct 30 13:27:00.149348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.148905    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"
Oct 30 13:27:00.149348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.148921    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9" event={"ID":"7371fc54-7fdf-4217-a7a7-4ea682e31616","Type":"ContainerDied","Data":"791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f"}
Oct 30 13:27:00.149348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.148989    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9" event={"ID":"7371fc54-7fdf-4217-a7a7-4ea682e31616","Type":"ContainerDied","Data":"0058030b6371076978adbf9500ed42ef891668b3eeaac05234864dadf97029e0"}
Oct 30 13:27:00.149348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.149019    4993 scope.go:122] "RemoveContainer" containerID="791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f"
Oct 30 13:27:00.164863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.164756    4993 scope.go:122] "RemoveContainer" containerID="791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f"
Oct 30 13:27:00.165178 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:27:00.165144    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f\": container with ID starting with 791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f not found: ID does not exist" containerID="791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f"
Oct 30 13:27:00.165310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.165189    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f"} err="failed to get container status \"791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f\": rpc error: code = NotFound desc = could not find container \"791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f\": container with ID starting with 791b61d7cff1cf483c227fa0522123c0aa2573a2eae94f084911018e4b96322f not found: ID does not exist"
Oct 30 13:27:00.207607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.207548    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"]
Oct 30 13:27:00.211977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:00.211935    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7gfgz9"]
Oct 30 13:27:01.691972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:01.691929    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7371fc54-7fdf-4217-a7a7-4ea682e31616" path="/var/lib/kubelet/pods/7371fc54-7fdf-4217-a7a7-4ea682e31616/volumes"
Oct 30 13:27:02.043369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:02.043339    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:27:06.348654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.348615    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"]
Oct 30 13:27:06.349273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.349254    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7371fc54-7fdf-4217-a7a7-4ea682e31616" containerName="payload-processing"
Oct 30 13:27:06.349327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.349274    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7371fc54-7fdf-4217-a7a7-4ea682e31616" containerName="payload-processing"
Oct 30 13:27:06.349375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.349365    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7371fc54-7fdf-4217-a7a7-4ea682e31616" containerName="payload-processing"
Oct 30 13:27:06.354355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.354330    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.382076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.382036    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"]
Oct 30 13:27:06.387515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.387470    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8740d4\""
Oct 30 13:27:06.387719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.387601    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"kube-root-ca.crt\""
Oct 30 13:27:06.387719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.387691    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"routing-overlay\""
Oct 30 13:27:06.387836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.387759    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"openshift-service-ca.crt\""
Oct 30 13:27:06.395720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.395579    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-5a382f35eb83\""
Oct 30 13:27:06.395918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.395753    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-d8628f05d549\""
Oct 30 13:27:06.395918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.395769    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-6e5458768ddf\""
Oct 30 13:27:06.395918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.395889    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-091b9769f7e9\""
Oct 30 13:27:06.396089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.395987    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-e2e--77992003-dockercfg-zp5z8\""
Oct 30 13:27:06.396240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.396173    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-a4617c99fc7d\""
Oct 30 13:27:06.445312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.445263    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.445536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.445324    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrg8h\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-kube-api-access-rrg8h\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.445536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.445449    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.445643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.445565    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.563601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.563556    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.580415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.580372    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrg8h\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-kube-api-access-rrg8h\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.580766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.580743    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.580953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.580937    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.586636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.565490    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.588041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.588006    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.592230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.589005    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.610361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.610282    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrg8h\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-kube-api-access-rrg8h\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.670668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.670622    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:06.985605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:06.985570    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"]
Oct 30 13:27:06.989027 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:27:06.988156    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6203c315_864d_4ede_94e5_19fbfe3650cf.slice/crio-bd47381a0126def706168a741e0c9e8941188275fafb45dbf3953f742be8f97c WatchSource:0}: Error finding container bd47381a0126def706168a741e0c9e8941188275fafb45dbf3953f742be8f97c: Status 404 returned error can't find the container with id bd47381a0126def706168a741e0c9e8941188275fafb45dbf3953f742be8f97c
Oct 30 13:27:07.288280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:07.288159    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm" event={"ID":"6203c315-864d-4ede-94e5-19fbfe3650cf","Type":"ContainerStarted","Data":"573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745"}
Oct 30 13:27:07.288451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:07.288284    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm" event={"ID":"6203c315-864d-4ede-94e5-19fbfe3650cf","Type":"ContainerStarted","Data":"bd47381a0126def706168a741e0c9e8941188275fafb45dbf3953f742be8f97c"}
Oct 30 13:27:07.288692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:07.288671    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:07.334760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:07.334696    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm" podStartSLOduration=1.334679715 podStartE2EDuration="1.334679715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:07.332412261 +0000 UTC m=+3291.808131914" watchObservedRunningTime="2026-10-30 13:27:07.334679715 +0000 UTC m=+3291.810399325"
Oct 30 13:27:08.570280 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:27:08.570241    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:27:16.284567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:16.284532    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:27:16.314373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:16.314342    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:27:18.301286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:18.301202    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:20.241799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:20.241764    4993 scope.go:122] "RemoveContainer" containerID="52a940c8994546871fd5cc34bd68dd99f444edbe0c52d52f75ae46a451a811aa"
Oct 30 13:27:38.685230 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:27:38.684680    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:27:50.476395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.476352    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"]
Oct 30 13:27:50.481348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.481309    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.497332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.497299    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:27:50.502834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.502793    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"]
Oct 30 13:27:50.508283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.508250    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:27:50.508479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.508302    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:27:50.508479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.508341    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:27:50.607334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.589384    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgk8\" (UniqueName: \"kubernetes.io/projected/87290992-a709-4f80-87ca-76311600e068-kube-api-access-flgk8\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.607334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.606133    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87290992-a709-4f80-87ca-76311600e068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.607334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.606412    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87290992-a709-4f80-87ca-76311600e068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.707837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.707794    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgk8\" (UniqueName: \"kubernetes.io/projected/87290992-a709-4f80-87ca-76311600e068-kube-api-access-flgk8\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.707837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.707840    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87290992-a709-4f80-87ca-76311600e068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.708407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.708298    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87290992-a709-4f80-87ca-76311600e068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.708911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.708885    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87290992-a709-4f80-87ca-76311600e068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.710381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.710357    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87290992-a709-4f80-87ca-76311600e068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.731325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.731285    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgk8\" (UniqueName: \"kubernetes.io/projected/87290992-a709-4f80-87ca-76311600e068-kube-api-access-flgk8\") pod \"maas-api-revoke-sub-580313b2-8w6m8\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:50.796910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:50.796864    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:51.068877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:51.068795    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"]
Oct 30 13:27:51.108585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:51.108556    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"]
Oct 30 13:27:51.111111 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:27:51.111078    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87290992_a709_4f80_87ca_76311600e068.slice/crio-2bfa79cb476b4463e98223ed0f14cc7afe012d1255c9c0b0a3d67e50363657f6 WatchSource:0}: Error finding container 2bfa79cb476b4463e98223ed0f14cc7afe012d1255c9c0b0a3d67e50363657f6: Status 404 returned error can't find the container with id 2bfa79cb476b4463e98223ed0f14cc7afe012d1255c9c0b0a3d67e50363657f6
Oct 30 13:27:51.760727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:51.760682    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8" event={"ID":"87290992-a709-4f80-87ca-76311600e068","Type":"ContainerStarted","Data":"fc35b64ddf36cc0c247cbd15f760a5e5d9293ea66a20a2187f68fc15c730122c"}
Oct 30 13:27:51.761296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:51.760749    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8" event={"ID":"87290992-a709-4f80-87ca-76311600e068","Type":"ContainerStarted","Data":"2bfa79cb476b4463e98223ed0f14cc7afe012d1255c9c0b0a3d67e50363657f6"}
Oct 30 13:27:51.826182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:51.826125    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8" podStartSLOduration=1.826106823 podStartE2EDuration="1.826106823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:51.823368182 +0000 UTC m=+3336.299087792" watchObservedRunningTime="2026-10-30 13:27:51.826106823 +0000 UTC m=+3336.301826432"
Oct 30 13:27:52.771841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:52.771798    4993 generic.go:356] "Generic (PLEG): container finished" podID="87290992-a709-4f80-87ca-76311600e068" containerID="fc35b64ddf36cc0c247cbd15f760a5e5d9293ea66a20a2187f68fc15c730122c" exitCode=0
Oct 30 13:27:52.772407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:52.771877    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8" event={"ID":"87290992-a709-4f80-87ca-76311600e068","Type":"ContainerDied","Data":"fc35b64ddf36cc0c247cbd15f760a5e5d9293ea66a20a2187f68fc15c730122c"}
Oct 30 13:27:53.956224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:53.956181    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:54.054614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.054517    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87290992-a709-4f80-87ca-76311600e068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87290992-a709-4f80-87ca-76311600e068-maas-api-xks-ca\") pod \"87290992-a709-4f80-87ca-76311600e068\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") "
Oct 30 13:27:54.054792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.054656    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87290992-a709-4f80-87ca-76311600e068-kube-api-access-flgk8\" (UniqueName: \"kubernetes.io/projected/87290992-a709-4f80-87ca-76311600e068-kube-api-access-flgk8\") pod \"87290992-a709-4f80-87ca-76311600e068\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") "
Oct 30 13:27:54.054792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.054750    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87290992-a709-4f80-87ca-76311600e068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87290992-a709-4f80-87ca-76311600e068-maas-api-openshift-ca\") pod \"87290992-a709-4f80-87ca-76311600e068\" (UID: \"87290992-a709-4f80-87ca-76311600e068\") "
Oct 30 13:27:54.055202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.055156    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87290992-a709-4f80-87ca-76311600e068-maas-api-openshift-ca" pod "87290992-a709-4f80-87ca-76311600e068" (UID: "87290992-a709-4f80-87ca-76311600e068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:54.055458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.055431    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87290992-a709-4f80-87ca-76311600e068-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:27:54.056861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.056836    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87290992-a709-4f80-87ca-76311600e068-maas-api-xks-ca" pod "87290992-a709-4f80-87ca-76311600e068" (UID: "87290992-a709-4f80-87ca-76311600e068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:54.057048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.057032    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87290992-a709-4f80-87ca-76311600e068-kube-api-access-flgk8" pod "87290992-a709-4f80-87ca-76311600e068" (UID: "87290992-a709-4f80-87ca-76311600e068"). InnerVolumeSpecName "kube-api-access-flgk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:54.167055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.161524    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87290992-a709-4f80-87ca-76311600e068-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:27:54.167055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.161572    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flgk8\" (UniqueName: \"kubernetes.io/projected/87290992-a709-4f80-87ca-76311600e068-kube-api-access-flgk8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:27:54.791466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.791435    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"
Oct 30 13:27:54.791466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.791454    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8" event={"ID":"87290992-a709-4f80-87ca-76311600e068","Type":"ContainerDied","Data":"2bfa79cb476b4463e98223ed0f14cc7afe012d1255c9c0b0a3d67e50363657f6"}
Oct 30 13:27:54.791741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:54.791511    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bfa79cb476b4463e98223ed0f14cc7afe012d1255c9c0b0a3d67e50363657f6"
Oct 30 13:27:56.069333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.069177    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm" podUID="6203c315-864d-4ede-94e5-19fbfe3650cf" containerName="payload-processing" containerID="cri-o://573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745" gracePeriod=25
Oct 30 13:27:56.361912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.361875    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:56.496530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.496491    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-routing-overlay-volume\") pod \"6203c315-864d-4ede-94e5-19fbfe3650cf\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") "
Oct 30 13:27:56.496737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.496567    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-kube-api-access-rrg8h\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-kube-api-access-rrg8h\") pod \"6203c315-864d-4ede-94e5-19fbfe3650cf\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") "
Oct 30 13:27:56.496737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.496714    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-plugins-config-volume\") pod \"6203c315-864d-4ede-94e5-19fbfe3650cf\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") "
Oct 30 13:27:56.496892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.496742    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-provider-credentials\") pod \"6203c315-864d-4ede-94e5-19fbfe3650cf\" (UID: \"6203c315-864d-4ede-94e5-19fbfe3650cf\") "
Oct 30 13:27:56.496955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.496918    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-routing-overlay-volume" pod "6203c315-864d-4ede-94e5-19fbfe3650cf" (UID: "6203c315-864d-4ede-94e5-19fbfe3650cf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:56.497195 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.497163    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-plugins-config-volume" pod "6203c315-864d-4ede-94e5-19fbfe3650cf" (UID: "6203c315-864d-4ede-94e5-19fbfe3650cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:56.497408 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.497386    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:27:56.497547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.497411    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6203c315-864d-4ede-94e5-19fbfe3650cf-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:27:56.499699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.499673    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-provider-credentials" pod "6203c315-864d-4ede-94e5-19fbfe3650cf" (UID: "6203c315-864d-4ede-94e5-19fbfe3650cf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:56.500331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.500306    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-kube-api-access-rrg8h" pod "6203c315-864d-4ede-94e5-19fbfe3650cf" (UID: "6203c315-864d-4ede-94e5-19fbfe3650cf"). InnerVolumeSpecName "kube-api-access-rrg8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:56.599531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.599430    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:27:56.599531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.599478    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrg8h\" (UniqueName: \"kubernetes.io/projected/6203c315-864d-4ede-94e5-19fbfe3650cf-kube-api-access-rrg8h\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:27:56.808742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.808700    4993 generic.go:356] "Generic (PLEG): container finished" podID="6203c315-864d-4ede-94e5-19fbfe3650cf" containerID="573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745" exitCode=0
Oct 30 13:27:56.808896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.808764    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"
Oct 30 13:27:56.808896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.808778    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm" event={"ID":"6203c315-864d-4ede-94e5-19fbfe3650cf","Type":"ContainerDied","Data":"573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745"}
Oct 30 13:27:56.808896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.808836    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm" event={"ID":"6203c315-864d-4ede-94e5-19fbfe3650cf","Type":"ContainerDied","Data":"bd47381a0126def706168a741e0c9e8941188275fafb45dbf3953f742be8f97c"}
Oct 30 13:27:56.808896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.808858    4993 scope.go:122] "RemoveContainer" containerID="573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745"
Oct 30 13:27:56.823493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.823468    4993 scope.go:122] "RemoveContainer" containerID="573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745"
Oct 30 13:27:56.823808 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:27:56.823786    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745\": container with ID starting with 573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745 not found: ID does not exist" containerID="573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745"
Oct 30 13:27:56.823892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.823816    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745"} err="failed to get container status \"573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745\": rpc error: code = NotFound desc = could not find container \"573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745\": container with ID starting with 573cca5e0a3fa5be120052beb2e428b46a9e7e986a485f16d126af6171082745 not found: ID does not exist"
Oct 30 13:27:56.863073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.862972    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"]
Oct 30 13:27:56.868889 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:56.868852    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-5b99n4jrm"]
Oct 30 13:27:57.694174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:27:57.694133    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6203c315-864d-4ede-94e5-19fbfe3650cf" path="/var/lib/kubelet/pods/6203c315-864d-4ede-94e5-19fbfe3650cf/volumes"
Oct 30 13:28:02.076267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.076226    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"]
Oct 30 13:28:02.076804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.076790    4993 cpu_manager.go:414] "containerMap: removing container" podUID="87290992-a709-4f80-87ca-76311600e068" containerName="revoke-keys"
Oct 30 13:28:02.076858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.076812    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87290992-a709-4f80-87ca-76311600e068" containerName="revoke-keys"
Oct 30 13:28:02.076899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.076860    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6203c315-864d-4ede-94e5-19fbfe3650cf" containerName="payload-processing"
Oct 30 13:28:02.076899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.076866    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6203c315-864d-4ede-94e5-19fbfe3650cf" containerName="payload-processing"
Oct 30 13:28:02.076982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.076950    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="87290992-a709-4f80-87ca-76311600e068" containerName="revoke-keys"
Oct 30 13:28:02.076982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.076969    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6203c315-864d-4ede-94e5-19fbfe3650cf" containerName="payload-processing"
Oct 30 13:28:02.081535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.081510    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.101253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.101222    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"]
Oct 30 13:28:02.108464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.108430    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:02.108663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.108489    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:02.108663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.108577    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:28:02.108942 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.108923    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:02.197115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.197067    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.197380 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.197278    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.197380 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.197325    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558nm\" (UniqueName: \"kubernetes.io/projected/bf277479-f049-4e98-84a2-3e4da9846796-kube-api-access-558nm\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.298694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.298658    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.298865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.298769    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.298865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.298807    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558nm\" (UniqueName: \"kubernetes.io/projected/bf277479-f049-4e98-84a2-3e4da9846796-kube-api-access-558nm\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.299494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.299465    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.301321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.301293    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.322702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.322661    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-558nm\" (UniqueName: \"kubernetes.io/projected/bf277479-f049-4e98-84a2-3e4da9846796-kube-api-access-558nm\") pod \"maas-api-revoke-sub-f51ec470-82bgl\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.397071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.396976    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:02.725943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.725578    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"]
Oct 30 13:28:02.729220 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:02.729158    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf277479_f049_4e98_84a2_3e4da9846796.slice/crio-e9684768e0a3b7890078003ed9905dbfea014def091992e9bff60e2c96f9c23c WatchSource:0}: Error finding container e9684768e0a3b7890078003ed9905dbfea014def091992e9bff60e2c96f9c23c: Status 404 returned error can't find the container with id e9684768e0a3b7890078003ed9905dbfea014def091992e9bff60e2c96f9c23c
Oct 30 13:28:02.921007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:02.920957    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl" event={"ID":"bf277479-f049-4e98-84a2-3e4da9846796","Type":"ContainerStarted","Data":"e9684768e0a3b7890078003ed9905dbfea014def091992e9bff60e2c96f9c23c"}
Oct 30 13:28:03.205469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.205384    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"]
Oct 30 13:28:03.210937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.210915    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.234964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.234924    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.235173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.235030    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wcj\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-kube-api-access-j2wcj\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.235294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.235241    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.235374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.235346    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.237434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.237409    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"]
Oct 30 13:28:03.244273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.244198    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-091b9769f7e9\""
Oct 30 13:28:03.244423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.244403    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-e2e--77992003-dockercfg-zp5z8\""
Oct 30 13:28:03.244508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.244454    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-a4617c99fc7d\""
Oct 30 13:28:03.244508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.244499    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-6e5458768ddf\""
Oct 30 13:28:03.244613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.244527    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"openshift-service-ca.crt\""
Oct 30 13:28:03.244613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.244601    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8740d4\""
Oct 30 13:28:03.244713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.244651    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"kube-root-ca.crt\""
Oct 30 13:28:03.244713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.244670    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"routing-overlay\""
Oct 30 13:28:03.336793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.336751    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wcj\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-kube-api-access-j2wcj\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.336983 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.336956    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.337071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.337056    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.337114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.337095    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.337855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.337827    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.337973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.337936    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.339932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.339906    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.360504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.360470    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wcj\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-kube-api-access-j2wcj\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.526152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.526107    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:03.833192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.833162    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"]
Oct 30 13:28:03.836027 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:03.835993    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fc2dca_52be_445a_b1df_ca95a05605f8.slice/crio-50b03b6dedc01e79b3db42b2d20375c6a8d15a1750b0378947c96422335e5e6b WatchSource:0}: Error finding container 50b03b6dedc01e79b3db42b2d20375c6a8d15a1750b0378947c96422335e5e6b: Status 404 returned error can't find the container with id 50b03b6dedc01e79b3db42b2d20375c6a8d15a1750b0378947c96422335e5e6b
Oct 30 13:28:03.930233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.930176    4993 generic.go:356] "Generic (PLEG): container finished" podID="bf277479-f049-4e98-84a2-3e4da9846796" containerID="3241458d78c963a763b650d63ab03937cda0ae17ee03081e2859f4198e0ea092" exitCode=0
Oct 30 13:28:03.930418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.930242    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl" event={"ID":"bf277479-f049-4e98-84a2-3e4da9846796","Type":"ContainerDied","Data":"3241458d78c963a763b650d63ab03937cda0ae17ee03081e2859f4198e0ea092"}
Oct 30 13:28:03.931833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:03.931802    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf" event={"ID":"69fc2dca-52be-445a-b1df-ca95a05605f8","Type":"ContainerStarted","Data":"50b03b6dedc01e79b3db42b2d20375c6a8d15a1750b0378947c96422335e5e6b"}
Oct 30 13:28:04.976687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:04.975889    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf" event={"ID":"69fc2dca-52be-445a-b1df-ca95a05605f8","Type":"ContainerStarted","Data":"d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86"}
Oct 30 13:28:04.976687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:04.976270    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:05.019993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.019922    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf" podStartSLOduration=2.019901576 podStartE2EDuration="2.019901576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:05.018075907 +0000 UTC m=+3349.493795517" watchObservedRunningTime="2026-10-30 13:28:05.019901576 +0000 UTC m=+3349.495621191"
Oct 30 13:28:05.155739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.155709    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:05.275079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.274980    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf277479-f049-4e98-84a2-3e4da9846796-kube-api-access-558nm\" (UniqueName: \"kubernetes.io/projected/bf277479-f049-4e98-84a2-3e4da9846796-kube-api-access-558nm\") pod \"bf277479-f049-4e98-84a2-3e4da9846796\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") "
Oct 30 13:28:05.275294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.275200    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-openshift-ca\") pod \"bf277479-f049-4e98-84a2-3e4da9846796\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") "
Oct 30 13:28:05.275378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.275338    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-xks-ca\") pod \"bf277479-f049-4e98-84a2-3e4da9846796\" (UID: \"bf277479-f049-4e98-84a2-3e4da9846796\") "
Oct 30 13:28:05.275674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.275644    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-openshift-ca" pod "bf277479-f049-4e98-84a2-3e4da9846796" (UID: "bf277479-f049-4e98-84a2-3e4da9846796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:05.275923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.275903    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:05.277412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.277375    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf277479-f049-4e98-84a2-3e4da9846796-kube-api-access-558nm" pod "bf277479-f049-4e98-84a2-3e4da9846796" (UID: "bf277479-f049-4e98-84a2-3e4da9846796"). InnerVolumeSpecName "kube-api-access-558nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:05.277540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.277444    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-xks-ca" pod "bf277479-f049-4e98-84a2-3e4da9846796" (UID: "bf277479-f049-4e98-84a2-3e4da9846796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:05.377287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.377244    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277479-f049-4e98-84a2-3e4da9846796-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:05.377287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.377285    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-558nm\" (UniqueName: \"kubernetes.io/projected/bf277479-f049-4e98-84a2-3e4da9846796-kube-api-access-558nm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:05.985319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.985285    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"
Oct 30 13:28:05.985865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.985436    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl" event={"ID":"bf277479-f049-4e98-84a2-3e4da9846796","Type":"ContainerDied","Data":"e9684768e0a3b7890078003ed9905dbfea014def091992e9bff60e2c96f9c23c"}
Oct 30 13:28:05.985865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:05.985476    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9684768e0a3b7890078003ed9905dbfea014def091992e9bff60e2c96f9c23c"
Oct 30 13:28:08.572130 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:28:08.572089    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:28:11.192786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.192740    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"]
Oct 30 13:28:11.193440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.193331    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bf277479-f049-4e98-84a2-3e4da9846796" containerName="revoke-keys"
Oct 30 13:28:11.193440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.193345    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf277479-f049-4e98-84a2-3e4da9846796" containerName="revoke-keys"
Oct 30 13:28:11.193574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.193484    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf277479-f049-4e98-84a2-3e4da9846796" containerName="revoke-keys"
Oct 30 13:28:11.196691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.196667    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.217743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.217710    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"]
Oct 30 13:28:11.224011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.223983    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:11.224192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.224132    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:28:11.224342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.224227    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:11.224463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.224425    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:11.256457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.255227    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.256457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.255544    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.256457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.256194    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l2rp\" (UniqueName: \"kubernetes.io/projected/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-kube-api-access-4l2rp\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.357879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.357836    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l2rp\" (UniqueName: \"kubernetes.io/projected/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-kube-api-access-4l2rp\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.358061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.357948    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.358117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.358080    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.358666 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.358636    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.360546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.360513    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.385241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.385189    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l2rp\" (UniqueName: \"kubernetes.io/projected/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-kube-api-access-4l2rp\") pod \"maas-api-revoke-sub-de73496d-zgpgv\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.511043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.511006    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:11.822793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:11.822760    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"]
Oct 30 13:28:11.825094 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:11.825064    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e3f1fa_135a_4f28_a97a_7f5fff516b6e.slice/crio-ea3f3fee5a31ffc7bc80213b9516686be7ed6c950a6cbe3e95bdff5f08ff52d0 WatchSource:0}: Error finding container ea3f3fee5a31ffc7bc80213b9516686be7ed6c950a6cbe3e95bdff5f08ff52d0: Status 404 returned error can't find the container with id ea3f3fee5a31ffc7bc80213b9516686be7ed6c950a6cbe3e95bdff5f08ff52d0
Oct 30 13:28:12.029794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.029750    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv" event={"ID":"78e3f1fa-135a-4f28-a97a-7f5fff516b6e","Type":"ContainerStarted","Data":"ea3f3fee5a31ffc7bc80213b9516686be7ed6c950a6cbe3e95bdff5f08ff52d0"}
Oct 30 13:28:12.816829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.816787    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"]
Oct 30 13:28:12.821260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.821233    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:12.840752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.840722    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"]
Oct 30 13:28:12.880032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.880002    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:12.880172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.880037    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74kj\" (UniqueName: \"kubernetes.io/projected/3688328f-acfd-4f9a-9a6f-43d3f6f66889-kube-api-access-b74kj\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:12.880295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.880189    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:12.981738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.981694    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:12.981738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.981741    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74kj\" (UniqueName: \"kubernetes.io/projected/3688328f-acfd-4f9a-9a6f-43d3f6f66889-kube-api-access-b74kj\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:12.981970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.981828    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:12.982562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.982539    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:12.984297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:12.984271    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:13.007701 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:13.007644    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74kj\" (UniqueName: \"kubernetes.io/projected/3688328f-acfd-4f9a-9a6f-43d3f6f66889-kube-api-access-b74kj\") pod \"maas-api-revoke-sub-dacada95-8vwmw\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:13.040902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:13.040864    4993 generic.go:356] "Generic (PLEG): container finished" podID="78e3f1fa-135a-4f28-a97a-7f5fff516b6e" containerID="4f82845ef0abd034c0cf6730a733d5f8990a181561b6d3dafac65cf10e89b481" exitCode=0
Oct 30 13:28:13.041117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:13.040947    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv" event={"ID":"78e3f1fa-135a-4f28-a97a-7f5fff516b6e","Type":"ContainerDied","Data":"4f82845ef0abd034c0cf6730a733d5f8990a181561b6d3dafac65cf10e89b481"}
Oct 30 13:28:13.138646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:13.138557    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:13.492741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:13.492711    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"]
Oct 30 13:28:13.495045 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:13.494999    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3688328f_acfd_4f9a_9a6f_43d3f6f66889.slice/crio-dac59b86efd6548eaed8e19e14e099973e0c867c1df17eb88aca62513ca5a9e8 WatchSource:0}: Error finding container dac59b86efd6548eaed8e19e14e099973e0c867c1df17eb88aca62513ca5a9e8: Status 404 returned error can't find the container with id dac59b86efd6548eaed8e19e14e099973e0c867c1df17eb88aca62513ca5a9e8
Oct 30 13:28:14.050715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.050605    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw" event={"ID":"3688328f-acfd-4f9a-9a6f-43d3f6f66889","Type":"ContainerStarted","Data":"9aa5a69b52fb1ebfd9b809a2f5f40c1bed813d89a08f1c44651a94a7f39c83ba"}
Oct 30 13:28:14.050715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.050677    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw" event={"ID":"3688328f-acfd-4f9a-9a6f-43d3f6f66889","Type":"ContainerStarted","Data":"dac59b86efd6548eaed8e19e14e099973e0c867c1df17eb88aca62513ca5a9e8"}
Oct 30 13:28:14.106897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.106825    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw" podStartSLOduration=2.106809082 podStartE2EDuration="2.106809082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:14.102028583 +0000 UTC m=+3358.577748192" watchObservedRunningTime="2026-10-30 13:28:14.106809082 +0000 UTC m=+3358.582528692"
Oct 30 13:28:14.252824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.252792    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:14.409978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.409940    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-kube-api-access-4l2rp\" (UniqueName: \"kubernetes.io/projected/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-kube-api-access-4l2rp\") pod \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") "
Oct 30 13:28:14.410160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.410006    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-xks-ca\") pod \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") "
Oct 30 13:28:14.410160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.410094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-openshift-ca\") pod \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\" (UID: \"78e3f1fa-135a-4f28-a97a-7f5fff516b6e\") "
Oct 30 13:28:14.410482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.410454    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-openshift-ca" pod "78e3f1fa-135a-4f28-a97a-7f5fff516b6e" (UID: "78e3f1fa-135a-4f28-a97a-7f5fff516b6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:14.410835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.410810    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.412792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.412762    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-xks-ca" pod "78e3f1fa-135a-4f28-a97a-7f5fff516b6e" (UID: "78e3f1fa-135a-4f28-a97a-7f5fff516b6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:14.412969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.412950    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-kube-api-access-4l2rp" pod "78e3f1fa-135a-4f28-a97a-7f5fff516b6e" (UID: "78e3f1fa-135a-4f28-a97a-7f5fff516b6e"). InnerVolumeSpecName "kube-api-access-4l2rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:14.512357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.512312    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l2rp\" (UniqueName: \"kubernetes.io/projected/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-kube-api-access-4l2rp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.512357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:14.512348    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e3f1fa-135a-4f28-a97a-7f5fff516b6e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:15.060264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:15.060229    4993 generic.go:356] "Generic (PLEG): container finished" podID="3688328f-acfd-4f9a-9a6f-43d3f6f66889" containerID="9aa5a69b52fb1ebfd9b809a2f5f40c1bed813d89a08f1c44651a94a7f39c83ba" exitCode=0
Oct 30 13:28:15.060696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:15.060329    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw" event={"ID":"3688328f-acfd-4f9a-9a6f-43d3f6f66889","Type":"ContainerDied","Data":"9aa5a69b52fb1ebfd9b809a2f5f40c1bed813d89a08f1c44651a94a7f39c83ba"}
Oct 30 13:28:15.062361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:15.062329    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"
Oct 30 13:28:15.062502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:15.062355    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv" event={"ID":"78e3f1fa-135a-4f28-a97a-7f5fff516b6e","Type":"ContainerDied","Data":"ea3f3fee5a31ffc7bc80213b9516686be7ed6c950a6cbe3e95bdff5f08ff52d0"}
Oct 30 13:28:15.062502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:15.062405    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3f3fee5a31ffc7bc80213b9516686be7ed6c950a6cbe3e95bdff5f08ff52d0"
Oct 30 13:28:15.989322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:15.989288    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:28:16.260585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.260519    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:16.337200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.337162    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-xks-ca\") pod \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") "
Oct 30 13:28:16.337408 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.337316    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3688328f-acfd-4f9a-9a6f-43d3f6f66889-kube-api-access-b74kj\" (UniqueName: \"kubernetes.io/projected/3688328f-acfd-4f9a-9a6f-43d3f6f66889-kube-api-access-b74kj\") pod \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") "
Oct 30 13:28:16.337408 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.337348    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-openshift-ca\") pod \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\" (UID: \"3688328f-acfd-4f9a-9a6f-43d3f6f66889\") "
Oct 30 13:28:16.337842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.337803    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-openshift-ca" pod "3688328f-acfd-4f9a-9a6f-43d3f6f66889" (UID: "3688328f-acfd-4f9a-9a6f-43d3f6f66889"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:16.337980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.337961    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:16.340498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.340474    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-xks-ca" pod "3688328f-acfd-4f9a-9a6f-43d3f6f66889" (UID: "3688328f-acfd-4f9a-9a6f-43d3f6f66889"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:16.341302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.341276    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3688328f-acfd-4f9a-9a6f-43d3f6f66889-kube-api-access-b74kj" pod "3688328f-acfd-4f9a-9a6f-43d3f6f66889" (UID: "3688328f-acfd-4f9a-9a6f-43d3f6f66889"). InnerVolumeSpecName "kube-api-access-b74kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:16.439658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.439618    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3688328f-acfd-4f9a-9a6f-43d3f6f66889-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:16.439658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:16.439651    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74kj\" (UniqueName: \"kubernetes.io/projected/3688328f-acfd-4f9a-9a6f-43d3f6f66889-kube-api-access-b74kj\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:17.082130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:17.082089    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"
Oct 30 13:28:17.082130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:17.082111    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw" event={"ID":"3688328f-acfd-4f9a-9a6f-43d3f6f66889","Type":"ContainerDied","Data":"dac59b86efd6548eaed8e19e14e099973e0c867c1df17eb88aca62513ca5a9e8"}
Oct 30 13:28:17.082473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:17.082174    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dac59b86efd6548eaed8e19e14e099973e0c867c1df17eb88aca62513ca5a9e8"
Oct 30 13:28:21.813232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.813181    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"]
Oct 30 13:28:21.813755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.813739    4993 cpu_manager.go:414] "containerMap: removing container" podUID="78e3f1fa-135a-4f28-a97a-7f5fff516b6e" containerName="revoke-keys"
Oct 30 13:28:21.813801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.813756    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e3f1fa-135a-4f28-a97a-7f5fff516b6e" containerName="revoke-keys"
Oct 30 13:28:21.813801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.813771    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3688328f-acfd-4f9a-9a6f-43d3f6f66889" containerName="revoke-keys"
Oct 30 13:28:21.813801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.813777    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3688328f-acfd-4f9a-9a6f-43d3f6f66889" containerName="revoke-keys"
Oct 30 13:28:21.813903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.813896    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e3f1fa-135a-4f28-a97a-7f5fff516b6e" containerName="revoke-keys"
Oct 30 13:28:21.813940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.813905    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3688328f-acfd-4f9a-9a6f-43d3f6f66889" containerName="revoke-keys"
Oct 30 13:28:21.818374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.818349    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:21.840437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.840408    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"]
Oct 30 13:28:21.841745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.841724    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:21.847356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.847328    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:21.847534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.847336    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:21.847609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.847441    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:28:21.907997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.907957    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:21.908152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.908108    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:21.908306 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:21.908282    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmq4b\" (UniqueName: \"kubernetes.io/projected/f706e66d-43ad-4c6e-b7dd-18a9031ed813-kube-api-access-vmq4b\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:22.009103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:22.009069    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmq4b\" (UniqueName: \"kubernetes.io/projected/f706e66d-43ad-4c6e-b7dd-18a9031ed813-kube-api-access-vmq4b\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:22.009328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:22.009215    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:22.009328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:22.009307    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:22.009991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:22.009965    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:22.011612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:22.011590    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:22.033148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:22.033119    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmq4b\" (UniqueName: \"kubernetes.io/projected/f706e66d-43ad-4c6e-b7dd-18a9031ed813-kube-api-access-vmq4b\") pod \"maas-api-revoke-sub-a33b7579-7xlqg\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:22.134645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:22.134551    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:22.459532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:22.459497    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"]
Oct 30 13:28:23.133493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:23.133398    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg" event={"ID":"f706e66d-43ad-4c6e-b7dd-18a9031ed813","Type":"ContainerStarted","Data":"7663b4e0180ae9fc8ff9e9eeed3f0a0aa394b9bf3bd7506064441b757ed69c2f"}
Oct 30 13:28:23.133493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:23.133470    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg" event={"ID":"f706e66d-43ad-4c6e-b7dd-18a9031ed813","Type":"ContainerStarted","Data":"541a07668a96bb2f0dea03332072909841f25a7c2473ca402ab39dc0878dd691"}
Oct 30 13:28:23.176423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:23.176360    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg" podStartSLOduration=2.176344321 podStartE2EDuration="2.176344321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:23.173035085 +0000 UTC m=+3367.648754695" watchObservedRunningTime="2026-10-30 13:28:23.176344321 +0000 UTC m=+3367.652063931"
Oct 30 13:28:24.142907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:24.142863    4993 generic.go:356] "Generic (PLEG): container finished" podID="f706e66d-43ad-4c6e-b7dd-18a9031ed813" containerID="7663b4e0180ae9fc8ff9e9eeed3f0a0aa394b9bf3bd7506064441b757ed69c2f" exitCode=0
Oct 30 13:28:24.143309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:24.142944    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg" event={"ID":"f706e66d-43ad-4c6e-b7dd-18a9031ed813","Type":"ContainerDied","Data":"7663b4e0180ae9fc8ff9e9eeed3f0a0aa394b9bf3bd7506064441b757ed69c2f"}
Oct 30 13:28:25.225832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.225798    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"]
Oct 30 13:28:25.230658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.230627    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.251028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.250982    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"]
Oct 30 13:28:25.334916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.334889    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:25.366259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.362988    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4ns\" (UniqueName: \"kubernetes.io/projected/90341b80-715d-44ea-87ff-e8e2f69b875e-kube-api-access-8l4ns\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.366259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.363260    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.366259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.363308    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.464434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.464346    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-xks-ca\") pod \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") "
Oct 30 13:28:25.464434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.464399    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f706e66d-43ad-4c6e-b7dd-18a9031ed813-kube-api-access-vmq4b\" (UniqueName: \"kubernetes.io/projected/f706e66d-43ad-4c6e-b7dd-18a9031ed813-kube-api-access-vmq4b\") pod \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") "
Oct 30 13:28:25.464680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.464536    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-openshift-ca\") pod \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\" (UID: \"f706e66d-43ad-4c6e-b7dd-18a9031ed813\") "
Oct 30 13:28:25.464874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.464843    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.465029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.464902    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.465230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.465117    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4ns\" (UniqueName: \"kubernetes.io/projected/90341b80-715d-44ea-87ff-e8e2f69b875e-kube-api-access-8l4ns\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.465980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.465572    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-openshift-ca" pod "f706e66d-43ad-4c6e-b7dd-18a9031ed813" (UID: "f706e66d-43ad-4c6e-b7dd-18a9031ed813"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:25.466356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.466333    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.467356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.467313    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f706e66d-43ad-4c6e-b7dd-18a9031ed813-kube-api-access-vmq4b" pod "f706e66d-43ad-4c6e-b7dd-18a9031ed813" (UID: "f706e66d-43ad-4c6e-b7dd-18a9031ed813"). InnerVolumeSpecName "kube-api-access-vmq4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:25.467503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.467473    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-xks-ca" pod "f706e66d-43ad-4c6e-b7dd-18a9031ed813" (UID: "f706e66d-43ad-4c6e-b7dd-18a9031ed813"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:25.467860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.467823    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.490503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.490468    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4ns\" (UniqueName: \"kubernetes.io/projected/90341b80-715d-44ea-87ff-e8e2f69b875e-kube-api-access-8l4ns\") pod \"maas-api-revoke-sub-f8f3cfdc-nwljc\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.551309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.551271    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:25.566394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.566366    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:25.566507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.566397    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmq4b\" (UniqueName: \"kubernetes.io/projected/f706e66d-43ad-4c6e-b7dd-18a9031ed813-kube-api-access-vmq4b\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:25.566507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.566407    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f706e66d-43ad-4c6e-b7dd-18a9031ed813-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:25.859435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:25.859386    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"]
Oct 30 13:28:25.863795 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:25.863761    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90341b80_715d_44ea_87ff_e8e2f69b875e.slice/crio-9590b2df083ea256e610d7306c069ab60fd0eb6d09d4947f4af8702f43388566 WatchSource:0}: Error finding container 9590b2df083ea256e610d7306c069ab60fd0eb6d09d4947f4af8702f43388566: Status 404 returned error can't find the container with id 9590b2df083ea256e610d7306c069ab60fd0eb6d09d4947f4af8702f43388566
Oct 30 13:28:26.162315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:26.162174    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"
Oct 30 13:28:26.162315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:26.162202    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg" event={"ID":"f706e66d-43ad-4c6e-b7dd-18a9031ed813","Type":"ContainerDied","Data":"541a07668a96bb2f0dea03332072909841f25a7c2473ca402ab39dc0878dd691"}
Oct 30 13:28:26.162315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:26.162296    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541a07668a96bb2f0dea03332072909841f25a7c2473ca402ab39dc0878dd691"
Oct 30 13:28:26.164101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:26.164062    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc" event={"ID":"90341b80-715d-44ea-87ff-e8e2f69b875e","Type":"ContainerStarted","Data":"9590b2df083ea256e610d7306c069ab60fd0eb6d09d4947f4af8702f43388566"}
Oct 30 13:28:27.179646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:27.179552    4993 generic.go:356] "Generic (PLEG): container finished" podID="90341b80-715d-44ea-87ff-e8e2f69b875e" containerID="c9ec658fa0e90f5956d794b3d53d4a657393c50c00f0e8335f435589495c1a54" exitCode=0
Oct 30 13:28:27.180073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:27.179639    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc" event={"ID":"90341b80-715d-44ea-87ff-e8e2f69b875e","Type":"ContainerDied","Data":"c9ec658fa0e90f5956d794b3d53d4a657393c50c00f0e8335f435589495c1a54"}
Oct 30 13:28:28.370357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.370317    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:28.405192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.405152    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-xks-ca\") pod \"90341b80-715d-44ea-87ff-e8e2f69b875e\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") "
Oct 30 13:28:28.405390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.405225    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90341b80-715d-44ea-87ff-e8e2f69b875e-kube-api-access-8l4ns\" (UniqueName: \"kubernetes.io/projected/90341b80-715d-44ea-87ff-e8e2f69b875e-kube-api-access-8l4ns\") pod \"90341b80-715d-44ea-87ff-e8e2f69b875e\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") "
Oct 30 13:28:28.405390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.405288    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-openshift-ca\") pod \"90341b80-715d-44ea-87ff-e8e2f69b875e\" (UID: \"90341b80-715d-44ea-87ff-e8e2f69b875e\") "
Oct 30 13:28:28.405699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.405676    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-openshift-ca" pod "90341b80-715d-44ea-87ff-e8e2f69b875e" (UID: "90341b80-715d-44ea-87ff-e8e2f69b875e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:28.405859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.405829    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.407635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.407601    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-xks-ca" pod "90341b80-715d-44ea-87ff-e8e2f69b875e" (UID: "90341b80-715d-44ea-87ff-e8e2f69b875e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:28.407791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.407633    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90341b80-715d-44ea-87ff-e8e2f69b875e-kube-api-access-8l4ns" pod "90341b80-715d-44ea-87ff-e8e2f69b875e" (UID: "90341b80-715d-44ea-87ff-e8e2f69b875e"). InnerVolumeSpecName "kube-api-access-8l4ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:28.507081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.507021    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90341b80-715d-44ea-87ff-e8e2f69b875e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:28.507081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:28.507057    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4ns\" (UniqueName: \"kubernetes.io/projected/90341b80-715d-44ea-87ff-e8e2f69b875e-kube-api-access-8l4ns\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:29.197185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:29.197147    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"
Oct 30 13:28:29.197185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:29.197168    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc" event={"ID":"90341b80-715d-44ea-87ff-e8e2f69b875e","Type":"ContainerDied","Data":"9590b2df083ea256e610d7306c069ab60fd0eb6d09d4947f4af8702f43388566"}
Oct 30 13:28:29.197502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:29.197246    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9590b2df083ea256e610d7306c069ab60fd0eb6d09d4947f4af8702f43388566"
Oct 30 13:28:34.255421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.255366    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"]
Oct 30 13:28:34.255954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.255942    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f706e66d-43ad-4c6e-b7dd-18a9031ed813" containerName="revoke-keys"
Oct 30 13:28:34.256031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.255955    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f706e66d-43ad-4c6e-b7dd-18a9031ed813" containerName="revoke-keys"
Oct 30 13:28:34.256031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.255977    4993 cpu_manager.go:414] "containerMap: removing container" podUID="90341b80-715d-44ea-87ff-e8e2f69b875e" containerName="revoke-keys"
Oct 30 13:28:34.256031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.255983    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90341b80-715d-44ea-87ff-e8e2f69b875e" containerName="revoke-keys"
Oct 30 13:28:34.256201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.256091    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="90341b80-715d-44ea-87ff-e8e2f69b875e" containerName="revoke-keys"
Oct 30 13:28:34.256201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.256113    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f706e66d-43ad-4c6e-b7dd-18a9031ed813" containerName="revoke-keys"
Oct 30 13:28:34.259291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.259270    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.280863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.280832    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"]
Oct 30 13:28:34.286359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.286332    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:34.286633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.286614    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:34.286753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.286704    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:34.292452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.292415    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:28:34.300155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.300115    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.300335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.300248    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.300531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.300510    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95f4h\" (UniqueName: \"kubernetes.io/projected/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-kube-api-access-95f4h\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.401911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.401865    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.402121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.401979    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.402254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.402200    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95f4h\" (UniqueName: \"kubernetes.io/projected/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-kube-api-access-95f4h\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.402744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.402715    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.404377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.404347    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.425946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.425916    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95f4h\" (UniqueName: \"kubernetes.io/projected/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-kube-api-access-95f4h\") pod \"maas-api-revoke-sub-23c6c411-f85tf\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.581703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.581598    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:34.897097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:34.897066    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"]
Oct 30 13:28:34.899792 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:34.899756    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc277ff81_ced3_4e2c_9cb6_896597a5d9bd.slice/crio-d21738f5a38dd1596564884ae153ae701b5924c1f9091319634b2fad6c7b8026 WatchSource:0}: Error finding container d21738f5a38dd1596564884ae153ae701b5924c1f9091319634b2fad6c7b8026: Status 404 returned error can't find the container with id d21738f5a38dd1596564884ae153ae701b5924c1f9091319634b2fad6c7b8026
Oct 30 13:28:35.248741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:35.248111    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf" event={"ID":"c277ff81-ced3-4e2c-9cb6-896597a5d9bd","Type":"ContainerStarted","Data":"d21738f5a38dd1596564884ae153ae701b5924c1f9091319634b2fad6c7b8026"}
Oct 30 13:28:36.257683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:36.257649    4993 generic.go:356] "Generic (PLEG): container finished" podID="c277ff81-ced3-4e2c-9cb6-896597a5d9bd" containerID="52ff4f7d402bcb131e16c86beeb2650335b087ece38230bbbc73600b8d3f092b" exitCode=0
Oct 30 13:28:36.258107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:36.257728    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf" event={"ID":"c277ff81-ced3-4e2c-9cb6-896597a5d9bd","Type":"ContainerDied","Data":"52ff4f7d402bcb131e16c86beeb2650335b087ece38230bbbc73600b8d3f092b"}
Oct 30 13:28:37.445730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.445699    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:37.690329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.670679    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-kube-api-access-95f4h\" (UniqueName: \"kubernetes.io/projected/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-kube-api-access-95f4h\") pod \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") "
Oct 30 13:28:37.690329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.670828    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-openshift-ca\") pod \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") "
Oct 30 13:28:37.690329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.670943    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-xks-ca\") pod \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\" (UID: \"c277ff81-ced3-4e2c-9cb6-896597a5d9bd\") "
Oct 30 13:28:37.714789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.714711    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-xks-ca" pod "c277ff81-ced3-4e2c-9cb6-896597a5d9bd" (UID: "c277ff81-ced3-4e2c-9cb6-896597a5d9bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:37.715123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.715099    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-openshift-ca" pod "c277ff81-ced3-4e2c-9cb6-896597a5d9bd" (UID: "c277ff81-ced3-4e2c-9cb6-896597a5d9bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:37.727440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.727390    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-kube-api-access-95f4h" pod "c277ff81-ced3-4e2c-9cb6-896597a5d9bd" (UID: "c277ff81-ced3-4e2c-9cb6-896597a5d9bd"). InnerVolumeSpecName "kube-api-access-95f4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:37.792464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.792428    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:37.792464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.792458    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:37.792464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:37.792469    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95f4h\" (UniqueName: \"kubernetes.io/projected/c277ff81-ced3-4e2c-9cb6-896597a5d9bd-kube-api-access-95f4h\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:38.277038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:38.277003    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"
Oct 30 13:28:38.277038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:38.277025    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf" event={"ID":"c277ff81-ced3-4e2c-9cb6-896597a5d9bd","Type":"ContainerDied","Data":"d21738f5a38dd1596564884ae153ae701b5924c1f9091319634b2fad6c7b8026"}
Oct 30 13:28:38.277297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:38.277073    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d21738f5a38dd1596564884ae153ae701b5924c1f9091319634b2fad6c7b8026"
Oct 30 13:28:38.568950 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:28:38.568864    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:28:41.861621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.861579    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"]
Oct 30 13:28:41.862169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.862153    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c277ff81-ced3-4e2c-9cb6-896597a5d9bd" containerName="revoke-keys"
Oct 30 13:28:41.862229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.862170    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c277ff81-ced3-4e2c-9cb6-896597a5d9bd" containerName="revoke-keys"
Oct 30 13:28:41.862310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.862300    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c277ff81-ced3-4e2c-9cb6-896597a5d9bd" containerName="revoke-keys"
Oct 30 13:28:41.865969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.865952    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:41.886257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.886196    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"]
Oct 30 13:28:41.893244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.892246    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:28:41.893244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.892264    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:41.893244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.892322    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:41.893244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:41.892334    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:42.051382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.051346    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510442de-7698-44a3-9c8c-accb239699df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.051580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.051482    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwczx\" (UniqueName: \"kubernetes.io/projected/510442de-7698-44a3-9c8c-accb239699df-kube-api-access-zwczx\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.051624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.051588    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510442de-7698-44a3-9c8c-accb239699df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.153180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.153088    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwczx\" (UniqueName: \"kubernetes.io/projected/510442de-7698-44a3-9c8c-accb239699df-kube-api-access-zwczx\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.153405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.153263    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510442de-7698-44a3-9c8c-accb239699df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.153405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.153381    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510442de-7698-44a3-9c8c-accb239699df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.154056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.154035    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510442de-7698-44a3-9c8c-accb239699df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.155844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.155820    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510442de-7698-44a3-9c8c-accb239699df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.176302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.176276    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwczx\" (UniqueName: \"kubernetes.io/projected/510442de-7698-44a3-9c8c-accb239699df-kube-api-access-zwczx\") pod \"maas-api-revoke-sub-8e101951-pcg9p\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.180113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.180089    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:42.502604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:42.502567    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"]
Oct 30 13:28:42.507636 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:42.507606    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510442de_7698_44a3_9c8c_accb239699df.slice/crio-980f7264ec8bb715c0948aabd2cccef45cbae0f428c0d325edc8765bb60b0603 WatchSource:0}: Error finding container 980f7264ec8bb715c0948aabd2cccef45cbae0f428c0d325edc8765bb60b0603: Status 404 returned error can't find the container with id 980f7264ec8bb715c0948aabd2cccef45cbae0f428c0d325edc8765bb60b0603
Oct 30 13:28:43.319560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:43.319522    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p" event={"ID":"510442de-7698-44a3-9c8c-accb239699df","Type":"ContainerStarted","Data":"d1d33a722992b151a103dd17b03013243d668b0c4333e3b49fcd264c8fb1a8e2"}
Oct 30 13:28:43.319996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:43.319582    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p" event={"ID":"510442de-7698-44a3-9c8c-accb239699df","Type":"ContainerStarted","Data":"980f7264ec8bb715c0948aabd2cccef45cbae0f428c0d325edc8765bb60b0603"}
Oct 30 13:28:43.366063 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:43.365981    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p" podStartSLOduration=2.365962185 podStartE2EDuration="2.365962185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:43.364510169 +0000 UTC m=+3387.840229780" watchObservedRunningTime="2026-10-30 13:28:43.365962185 +0000 UTC m=+3387.841681796"
Oct 30 13:28:44.329200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:44.329160    4993 generic.go:356] "Generic (PLEG): container finished" podID="510442de-7698-44a3-9c8c-accb239699df" containerID="d1d33a722992b151a103dd17b03013243d668b0c4333e3b49fcd264c8fb1a8e2" exitCode=0
Oct 30 13:28:44.329779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:44.329263    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p" event={"ID":"510442de-7698-44a3-9c8c-accb239699df","Type":"ContainerDied","Data":"d1d33a722992b151a103dd17b03013243d668b0c4333e3b49fcd264c8fb1a8e2"}
Oct 30 13:28:45.509639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.509613    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:45.520606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.520575    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510442de-7698-44a3-9c8c-accb239699df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510442de-7698-44a3-9c8c-accb239699df-maas-api-openshift-ca\") pod \"510442de-7698-44a3-9c8c-accb239699df\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") "
Oct 30 13:28:45.520767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.520614    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510442de-7698-44a3-9c8c-accb239699df-kube-api-access-zwczx\" (UniqueName: \"kubernetes.io/projected/510442de-7698-44a3-9c8c-accb239699df-kube-api-access-zwczx\") pod \"510442de-7698-44a3-9c8c-accb239699df\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") "
Oct 30 13:28:45.520767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.520642    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510442de-7698-44a3-9c8c-accb239699df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510442de-7698-44a3-9c8c-accb239699df-maas-api-xks-ca\") pod \"510442de-7698-44a3-9c8c-accb239699df\" (UID: \"510442de-7698-44a3-9c8c-accb239699df\") "
Oct 30 13:28:45.520922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.520901    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510442de-7698-44a3-9c8c-accb239699df-maas-api-openshift-ca" pod "510442de-7698-44a3-9c8c-accb239699df" (UID: "510442de-7698-44a3-9c8c-accb239699df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:45.521129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.521115    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510442de-7698-44a3-9c8c-accb239699df-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:45.522806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.522776    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510442de-7698-44a3-9c8c-accb239699df-maas-api-xks-ca" pod "510442de-7698-44a3-9c8c-accb239699df" (UID: "510442de-7698-44a3-9c8c-accb239699df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:45.522927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.522861    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510442de-7698-44a3-9c8c-accb239699df-kube-api-access-zwczx" pod "510442de-7698-44a3-9c8c-accb239699df" (UID: "510442de-7698-44a3-9c8c-accb239699df"). InnerVolumeSpecName "kube-api-access-zwczx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:45.622584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.622498    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwczx\" (UniqueName: \"kubernetes.io/projected/510442de-7698-44a3-9c8c-accb239699df-kube-api-access-zwczx\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:45.622584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:45.622537    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510442de-7698-44a3-9c8c-accb239699df-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:46.327846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.327804    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"]
Oct 30 13:28:46.362235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.344331    4993 cpu_manager.go:414] "containerMap: removing container" podUID="510442de-7698-44a3-9c8c-accb239699df" containerName="revoke-keys"
Oct 30 13:28:46.362513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.362485    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510442de-7698-44a3-9c8c-accb239699df" containerName="revoke-keys"
Oct 30 13:28:46.362880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.362861    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="510442de-7698-44a3-9c8c-accb239699df" containerName="revoke-keys"
Oct 30 13:28:46.415328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.381890    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.427653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.427613    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p" event={"ID":"510442de-7698-44a3-9c8c-accb239699df","Type":"ContainerDied","Data":"980f7264ec8bb715c0948aabd2cccef45cbae0f428c0d325edc8765bb60b0603"}
Oct 30 13:28:46.427932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.427912    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="980f7264ec8bb715c0948aabd2cccef45cbae0f428c0d325edc8765bb60b0603"
Oct 30 13:28:46.428112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.428099    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"
Oct 30 13:28:46.494959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.494900    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"]
Oct 30 13:28:46.556660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.556619    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vqcg\" (UniqueName: \"kubernetes.io/projected/93eeb04b-6c30-467c-a721-ceb32f9c086e-kube-api-access-9vqcg\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.557100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.556900    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.557100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.557037    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.658747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.658646    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.658942 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.658793    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqcg\" (UniqueName: \"kubernetes.io/projected/93eeb04b-6c30-467c-a721-ceb32f9c086e-kube-api-access-9vqcg\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.659019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.658983    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.659800 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.659774    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.661242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.661178    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.684152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.684122    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqcg\" (UniqueName: \"kubernetes.io/projected/93eeb04b-6c30-467c-a721-ceb32f9c086e-kube-api-access-9vqcg\") pod \"maas-api-revoke-sub-1f48b40c-wmk7l\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:46.784147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:46.784102    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:47.095876 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:47.095842    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93eeb04b_6c30_467c_a721_ceb32f9c086e.slice/crio-74e23862bbae367fe21381952505fd37ee7dbc1430488527eae03ee20b5b03f3 WatchSource:0}: Error finding container 74e23862bbae367fe21381952505fd37ee7dbc1430488527eae03ee20b5b03f3: Status 404 returned error can't find the container with id 74e23862bbae367fe21381952505fd37ee7dbc1430488527eae03ee20b5b03f3
Oct 30 13:28:47.096102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:47.096075    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"]
Oct 30 13:28:47.436900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:47.436859    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l" event={"ID":"93eeb04b-6c30-467c-a721-ceb32f9c086e","Type":"ContainerStarted","Data":"8a36f6b0b31905748cfd51b70b45441b59e0ac64b39cdc249590a87765e5e669"}
Oct 30 13:28:47.437142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:47.436925    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l" event={"ID":"93eeb04b-6c30-467c-a721-ceb32f9c086e","Type":"ContainerStarted","Data":"74e23862bbae367fe21381952505fd37ee7dbc1430488527eae03ee20b5b03f3"}
Oct 30 13:28:47.479648 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:47.479594    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l" podStartSLOduration=1.479577655 podStartE2EDuration="1.479577655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:47.476277922 +0000 UTC m=+3391.951997532" watchObservedRunningTime="2026-10-30 13:28:47.479577655 +0000 UTC m=+3391.955297264"
Oct 30 13:28:48.445651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:48.445613    4993 generic.go:356] "Generic (PLEG): container finished" podID="93eeb04b-6c30-467c-a721-ceb32f9c086e" containerID="8a36f6b0b31905748cfd51b70b45441b59e0ac64b39cdc249590a87765e5e669" exitCode=0
Oct 30 13:28:48.446434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:48.445702    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l" event={"ID":"93eeb04b-6c30-467c-a721-ceb32f9c086e","Type":"ContainerDied","Data":"8a36f6b0b31905748cfd51b70b45441b59e0ac64b39cdc249590a87765e5e669"}
Oct 30 13:28:49.697958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.697924    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:49.808395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.808352    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-xks-ca\") pod \"93eeb04b-6c30-467c-a721-ceb32f9c086e\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") "
Oct 30 13:28:49.808617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.808447    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-openshift-ca\") pod \"93eeb04b-6c30-467c-a721-ceb32f9c086e\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") "
Oct 30 13:28:49.808617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.808510    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93eeb04b-6c30-467c-a721-ceb32f9c086e-kube-api-access-9vqcg\" (UniqueName: \"kubernetes.io/projected/93eeb04b-6c30-467c-a721-ceb32f9c086e-kube-api-access-9vqcg\") pod \"93eeb04b-6c30-467c-a721-ceb32f9c086e\" (UID: \"93eeb04b-6c30-467c-a721-ceb32f9c086e\") "
Oct 30 13:28:49.808830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.808806    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-openshift-ca" pod "93eeb04b-6c30-467c-a721-ceb32f9c086e" (UID: "93eeb04b-6c30-467c-a721-ceb32f9c086e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:49.810751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.810715    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-xks-ca" pod "93eeb04b-6c30-467c-a721-ceb32f9c086e" (UID: "93eeb04b-6c30-467c-a721-ceb32f9c086e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:49.810882 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.810849    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93eeb04b-6c30-467c-a721-ceb32f9c086e-kube-api-access-9vqcg" pod "93eeb04b-6c30-467c-a721-ceb32f9c086e" (UID: "93eeb04b-6c30-467c-a721-ceb32f9c086e"). InnerVolumeSpecName "kube-api-access-9vqcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:49.910598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.910546    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vqcg\" (UniqueName: \"kubernetes.io/projected/93eeb04b-6c30-467c-a721-ceb32f9c086e-kube-api-access-9vqcg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:49.910598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.910584    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:49.910598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:49.910594    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93eeb04b-6c30-467c-a721-ceb32f9c086e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:50.483395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:50.483366    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"
Oct 30 13:28:50.483618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:50.483401    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l" event={"ID":"93eeb04b-6c30-467c-a721-ceb32f9c086e","Type":"ContainerDied","Data":"74e23862bbae367fe21381952505fd37ee7dbc1430488527eae03ee20b5b03f3"}
Oct 30 13:28:50.483618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:50.483460    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e23862bbae367fe21381952505fd37ee7dbc1430488527eae03ee20b5b03f3"
Oct 30 13:28:54.389229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:54.387403    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"]
Oct 30 13:28:58.500589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.500553    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"]
Oct 30 13:28:58.502116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.502091    4993 cpu_manager.go:414] "containerMap: removing container" podUID="93eeb04b-6c30-467c-a721-ceb32f9c086e" containerName="revoke-keys"
Oct 30 13:28:58.502328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.502316    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93eeb04b-6c30-467c-a721-ceb32f9c086e" containerName="revoke-keys"
Oct 30 13:28:58.503256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.503236    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="93eeb04b-6c30-467c-a721-ceb32f9c086e" containerName="revoke-keys"
Oct 30 13:28:58.534406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.533445    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"]
Oct 30 13:28:58.534406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.533578    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.548388 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.548358    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:58.548939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.548917    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:28:58.560407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.560378    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:58.560582 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.560469    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:58.687143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.687101    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.687352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.687198    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdzt\" (UniqueName: \"kubernetes.io/projected/6f40961f-88bc-45c8-8ff0-876a52e1353d-kube-api-access-gzdzt\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.687489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.687461    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.788529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.788430    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.788696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.788529    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdzt\" (UniqueName: \"kubernetes.io/projected/6f40961f-88bc-45c8-8ff0-876a52e1353d-kube-api-access-gzdzt\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.788750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.788701    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.789271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.789239    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.791300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.791277    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.812150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.812115    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdzt\" (UniqueName: \"kubernetes.io/projected/6f40961f-88bc-45c8-8ff0-876a52e1353d-kube-api-access-gzdzt\") pod \"maas-api-revoke-sub-b18996bd-xrsvd\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:58.851643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:58.851600    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:28:59.195799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.195770    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"]
Oct 30 13:28:59.199095 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:28:59.199049    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f40961f_88bc_45c8_8ff0_876a52e1353d.slice/crio-f43564f139cd6cd382c70810648a8969a948de864fb383894ef1f96d53329629 WatchSource:0}: Error finding container f43564f139cd6cd382c70810648a8969a948de864fb383894ef1f96d53329629: Status 404 returned error can't find the container with id f43564f139cd6cd382c70810648a8969a948de864fb383894ef1f96d53329629
Oct 30 13:28:59.389620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.389363    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9" podUID="05c8f749-e8b0-437f-9392-f9ee2bebad97" containerName="payload-processing" containerID="cri-o://e4e73bca748c8537ef731276e2ac4ce0a95aa787d8e1d9305c76d37297ed7e41" gracePeriod=25
Oct 30 13:28:59.470395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.470296    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"]
Oct 30 13:28:59.475533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.475509    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.494235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.493972    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"]
Oct 30 13:28:59.502231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.501008    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.502231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.501053    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5drtc\" (UniqueName: \"kubernetes.io/projected/ee72959e-de1a-4721-a621-8ccdad306a01-kube-api-access-5drtc\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.502231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.501160    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.603107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.603070    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.603107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.603117    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drtc\" (UniqueName: \"kubernetes.io/projected/ee72959e-de1a-4721-a621-8ccdad306a01-kube-api-access-5drtc\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.603375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.603256    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.603942 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.603912    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.605991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.605965    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.627162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.627118    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drtc\" (UniqueName: \"kubernetes.io/projected/ee72959e-de1a-4721-a621-8ccdad306a01-kube-api-access-5drtc\") pod \"maas-api-revoke-sub-f8721350-8v5mr\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:28:59.651087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.651050    4993 generic.go:356] "Generic (PLEG): container finished" podID="05c8f749-e8b0-437f-9392-f9ee2bebad97" containerID="e4e73bca748c8537ef731276e2ac4ce0a95aa787d8e1d9305c76d37297ed7e41" exitCode=0
Oct 30 13:28:59.651327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.651126    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9" event={"ID":"05c8f749-e8b0-437f-9392-f9ee2bebad97","Type":"ContainerDied","Data":"e4e73bca748c8537ef731276e2ac4ce0a95aa787d8e1d9305c76d37297ed7e41"}
Oct 30 13:28:59.652953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.652926    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd" event={"ID":"6f40961f-88bc-45c8-8ff0-876a52e1353d","Type":"ContainerStarted","Data":"73b454ff51323a2df1e7c0bb68dec2894ca7f8655282f475073bfc99d2d000ec"}
Oct 30 13:28:59.653073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.652973    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd" event={"ID":"6f40961f-88bc-45c8-8ff0-876a52e1353d","Type":"ContainerStarted","Data":"f43564f139cd6cd382c70810648a8969a948de864fb383894ef1f96d53329629"}
Oct 30 13:28:59.675971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.675941    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:28:59.695571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.695522    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd" podStartSLOduration=1.695507644 podStartE2EDuration="1.695507644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:59.693165487 +0000 UTC m=+3404.168885096" watchObservedRunningTime="2026-10-30 13:28:59.695507644 +0000 UTC m=+3404.171227254"
Oct 30 13:28:59.704587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.704559    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-provider-credentials\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-provider-credentials\") pod \"05c8f749-e8b0-437f-9392-f9ee2bebad97\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") "
Oct 30 13:28:59.704750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.704734    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-routing-overlay-volume\") pod \"05c8f749-e8b0-437f-9392-f9ee2bebad97\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") "
Oct 30 13:28:59.704815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.704791    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-plugins-config-volume\") pod \"05c8f749-e8b0-437f-9392-f9ee2bebad97\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") "
Oct 30 13:28:59.704884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.704845    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-kube-api-access-fgn49\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-kube-api-access-fgn49\") pod \"05c8f749-e8b0-437f-9392-f9ee2bebad97\" (UID: \"05c8f749-e8b0-437f-9392-f9ee2bebad97\") "
Oct 30 13:28:59.705125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.705099    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-routing-overlay-volume" pod "05c8f749-e8b0-437f-9392-f9ee2bebad97" (UID: "05c8f749-e8b0-437f-9392-f9ee2bebad97"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:59.705240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.705141    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-plugins-config-volume" pod "05c8f749-e8b0-437f-9392-f9ee2bebad97" (UID: "05c8f749-e8b0-437f-9392-f9ee2bebad97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:59.705632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.705607    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:59.705715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.705637    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c8f749-e8b0-437f-9392-f9ee2bebad97-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:59.707663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.707638    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-provider-credentials" pod "05c8f749-e8b0-437f-9392-f9ee2bebad97" (UID: "05c8f749-e8b0-437f-9392-f9ee2bebad97"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:59.707778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.707686    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-kube-api-access-fgn49" pod "05c8f749-e8b0-437f-9392-f9ee2bebad97" (UID: "05c8f749-e8b0-437f-9392-f9ee2bebad97"). InnerVolumeSpecName "kube-api-access-fgn49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:59.807006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.806971    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgn49\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-kube-api-access-fgn49\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:59.807006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.807009    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05c8f749-e8b0-437f-9392-f9ee2bebad97-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:28:59.820809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:28:59.820778    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:29:00.133636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.133551    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"]
Oct 30 13:29:00.138654 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:00.138620    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee72959e_de1a_4721_a621_8ccdad306a01.slice/crio-db71fd8732febb773e21dad74c8969d8a4bdb2a45379ac4f9ef158d0b9056673 WatchSource:0}: Error finding container db71fd8732febb773e21dad74c8969d8a4bdb2a45379ac4f9ef158d0b9056673: Status 404 returned error can't find the container with id db71fd8732febb773e21dad74c8969d8a4bdb2a45379ac4f9ef158d0b9056673
Oct 30 13:29:00.710232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.701606    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9" event={"ID":"05c8f749-e8b0-437f-9392-f9ee2bebad97","Type":"ContainerDied","Data":"02f8af64b4c271115d6bbba657529b899b4b6688773182e6ce2d987bc11be04b"}
Oct 30 13:29:00.710232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.701743    4993 scope.go:122] "RemoveContainer" containerID="e4e73bca748c8537ef731276e2ac4ce0a95aa787d8e1d9305c76d37297ed7e41"
Oct 30 13:29:00.710232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.701959    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"
Oct 30 13:29:00.755233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.740146    4993 generic.go:356] "Generic (PLEG): container finished" podID="6f40961f-88bc-45c8-8ff0-876a52e1353d" containerID="73b454ff51323a2df1e7c0bb68dec2894ca7f8655282f475073bfc99d2d000ec" exitCode=0
Oct 30 13:29:00.755233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.740370    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd" event={"ID":"6f40961f-88bc-45c8-8ff0-876a52e1353d","Type":"ContainerDied","Data":"73b454ff51323a2df1e7c0bb68dec2894ca7f8655282f475073bfc99d2d000ec"}
Oct 30 13:29:00.755233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.752368    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr" event={"ID":"ee72959e-de1a-4721-a621-8ccdad306a01","Type":"ContainerStarted","Data":"5e30eaae52ab867cd8fb535b40b42778e0f29e03fafd5c3487818425bfa97f6a"}
Oct 30 13:29:00.755233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.752438    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr" event={"ID":"ee72959e-de1a-4721-a621-8ccdad306a01","Type":"ContainerStarted","Data":"db71fd8732febb773e21dad74c8969d8a4bdb2a45379ac4f9ef158d0b9056673"}
Oct 30 13:29:00.893233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.892532    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr" podStartSLOduration=1.892511405 podStartE2EDuration="1.892511405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:00.862626564 +0000 UTC m=+3405.338346190" watchObservedRunningTime="2026-10-30 13:29:00.892511405 +0000 UTC m=+3405.368231016"
Oct 30 13:29:01.000048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:00.998163    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"]
Oct 30 13:29:01.024273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:01.001420    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-55c848tg9"]
Oct 30 13:29:01.696040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:01.695947    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c8f749-e8b0-437f-9392-f9ee2bebad97" path="/var/lib/kubelet/pods/05c8f749-e8b0-437f-9392-f9ee2bebad97/volumes"
Oct 30 13:29:01.765259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:01.765198    4993 generic.go:356] "Generic (PLEG): container finished" podID="ee72959e-de1a-4721-a621-8ccdad306a01" containerID="5e30eaae52ab867cd8fb535b40b42778e0f29e03fafd5c3487818425bfa97f6a" exitCode=0
Oct 30 13:29:01.765773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:01.765262    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr" event={"ID":"ee72959e-de1a-4721-a621-8ccdad306a01","Type":"ContainerDied","Data":"5e30eaae52ab867cd8fb535b40b42778e0f29e03fafd5c3487818425bfa97f6a"}
Oct 30 13:29:01.991156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:01.991118    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:29:02.061957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.061916    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-xks-ca\") pod \"6f40961f-88bc-45c8-8ff0-876a52e1353d\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") "
Oct 30 13:29:02.062714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.062305    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f40961f-88bc-45c8-8ff0-876a52e1353d-kube-api-access-gzdzt\" (UniqueName: \"kubernetes.io/projected/6f40961f-88bc-45c8-8ff0-876a52e1353d-kube-api-access-gzdzt\") pod \"6f40961f-88bc-45c8-8ff0-876a52e1353d\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") "
Oct 30 13:29:02.062959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.062934    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-openshift-ca\") pod \"6f40961f-88bc-45c8-8ff0-876a52e1353d\" (UID: \"6f40961f-88bc-45c8-8ff0-876a52e1353d\") "
Oct 30 13:29:02.063837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.063800    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-openshift-ca" pod "6f40961f-88bc-45c8-8ff0-876a52e1353d" (UID: "6f40961f-88bc-45c8-8ff0-876a52e1353d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:02.064152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.064123    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:02.065159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.065133    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f40961f-88bc-45c8-8ff0-876a52e1353d-kube-api-access-gzdzt" pod "6f40961f-88bc-45c8-8ff0-876a52e1353d" (UID: "6f40961f-88bc-45c8-8ff0-876a52e1353d"). InnerVolumeSpecName "kube-api-access-gzdzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:02.084604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.074927    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-xks-ca" pod "6f40961f-88bc-45c8-8ff0-876a52e1353d" (UID: "6f40961f-88bc-45c8-8ff0-876a52e1353d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:02.165853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.165809    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzdzt\" (UniqueName: \"kubernetes.io/projected/6f40961f-88bc-45c8-8ff0-876a52e1353d-kube-api-access-gzdzt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:02.165853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.165847    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f40961f-88bc-45c8-8ff0-876a52e1353d-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:02.464033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.463991    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"]
Oct 30 13:29:02.464872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.464847    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6f40961f-88bc-45c8-8ff0-876a52e1353d" containerName="revoke-keys"
Oct 30 13:29:02.464946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.464876    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f40961f-88bc-45c8-8ff0-876a52e1353d" containerName="revoke-keys"
Oct 30 13:29:02.464946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.464918    4993 cpu_manager.go:414] "containerMap: removing container" podUID="05c8f749-e8b0-437f-9392-f9ee2bebad97" containerName="payload-processing"
Oct 30 13:29:02.464946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.464926    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c8f749-e8b0-437f-9392-f9ee2bebad97" containerName="payload-processing"
Oct 30 13:29:02.465106 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.465086    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f40961f-88bc-45c8-8ff0-876a52e1353d" containerName="revoke-keys"
Oct 30 13:29:02.465161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.465107    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c8f749-e8b0-437f-9392-f9ee2bebad97" containerName="payload-processing"
Oct 30 13:29:02.469445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.469412    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.493824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.493775    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"]
Oct 30 13:29:02.496499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.496462    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"agc-cred-d2752895cbe6\""
Oct 30 13:29:02.571576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.571528    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btm5k\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-kube-api-access-btm5k\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.571758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.571729    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.571861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.571844    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.571929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.571916    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.673438 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.673395    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btm5k\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-kube-api-access-btm5k\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.673663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.673507    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.673663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.673569    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.673663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.673613    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.674377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.674345    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.674684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.674658    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.676611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.676584    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.698829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.698794    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btm5k\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-kube-api-access-btm5k\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.775091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.775049    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd" event={"ID":"6f40961f-88bc-45c8-8ff0-876a52e1353d","Type":"ContainerDied","Data":"f43564f139cd6cd382c70810648a8969a948de864fb383894ef1f96d53329629"}
Oct 30 13:29:02.775572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.775113    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f43564f139cd6cd382c70810648a8969a948de864fb383894ef1f96d53329629"
Oct 30 13:29:02.775572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.775058    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"
Oct 30 13:29:02.803783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.803746    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:02.929347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:02.929316    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:29:03.098287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.090288    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-openshift-ca\") pod \"ee72959e-de1a-4721-a621-8ccdad306a01\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") "
Oct 30 13:29:03.098287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.090375    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee72959e-de1a-4721-a621-8ccdad306a01-kube-api-access-5drtc\" (UniqueName: \"kubernetes.io/projected/ee72959e-de1a-4721-a621-8ccdad306a01-kube-api-access-5drtc\") pod \"ee72959e-de1a-4721-a621-8ccdad306a01\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") "
Oct 30 13:29:03.098287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.090651    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-xks-ca\") pod \"ee72959e-de1a-4721-a621-8ccdad306a01\" (UID: \"ee72959e-de1a-4721-a621-8ccdad306a01\") "
Oct 30 13:29:03.098287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.091999    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-openshift-ca" pod "ee72959e-de1a-4721-a621-8ccdad306a01" (UID: "ee72959e-de1a-4721-a621-8ccdad306a01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:03.102650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.101070    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee72959e-de1a-4721-a621-8ccdad306a01-kube-api-access-5drtc" pod "ee72959e-de1a-4721-a621-8ccdad306a01" (UID: "ee72959e-de1a-4721-a621-8ccdad306a01"). InnerVolumeSpecName "kube-api-access-5drtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:03.105583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.105543    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-xks-ca" pod "ee72959e-de1a-4721-a621-8ccdad306a01" (UID: "ee72959e-de1a-4721-a621-8ccdad306a01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:03.148458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.148422    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"]
Oct 30 13:29:03.150031 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:03.149967    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1532e5a1_3417_487b_95b4_947fb71b20a4.slice/crio-a76af8b2d46ad035edab00737f255dc029ba21ec825dc2257bfc901dc1226255 WatchSource:0}: Error finding container a76af8b2d46ad035edab00737f255dc029ba21ec825dc2257bfc901dc1226255: Status 404 returned error can't find the container with id a76af8b2d46ad035edab00737f255dc029ba21ec825dc2257bfc901dc1226255
Oct 30 13:29:03.192898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.192856    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:03.192898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.192890    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee72959e-de1a-4721-a621-8ccdad306a01-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:03.192898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.192904    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5drtc\" (UniqueName: \"kubernetes.io/projected/ee72959e-de1a-4721-a621-8ccdad306a01-kube-api-access-5drtc\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:03.786016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.785972    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6" event={"ID":"1532e5a1-3417-487b-95b4-947fb71b20a4","Type":"ContainerStarted","Data":"4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d"}
Oct 30 13:29:03.786549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.786044    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6" event={"ID":"1532e5a1-3417-487b-95b4-947fb71b20a4","Type":"ContainerStarted","Data":"a76af8b2d46ad035edab00737f255dc029ba21ec825dc2257bfc901dc1226255"}
Oct 30 13:29:03.786549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.786075    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:03.788080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.788048    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr" event={"ID":"ee72959e-de1a-4721-a621-8ccdad306a01","Type":"ContainerDied","Data":"db71fd8732febb773e21dad74c8969d8a4bdb2a45379ac4f9ef158d0b9056673"}
Oct 30 13:29:03.788429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.788083    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"
Oct 30 13:29:03.788429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.788101    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db71fd8732febb773e21dad74c8969d8a4bdb2a45379ac4f9ef158d0b9056673"
Oct 30 13:29:03.831111 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:03.831035    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6" podStartSLOduration=1.831015212 podStartE2EDuration="1.831015212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:03.828599638 +0000 UTC m=+3408.304319260" watchObservedRunningTime="2026-10-30 13:29:03.831015212 +0000 UTC m=+3408.306734824"
Oct 30 13:29:08.573022 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:29:08.572976    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:29:10.564219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:10.564167    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"]
Oct 30 13:29:11.009865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.009811    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"]
Oct 30 13:29:11.011300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.010759    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ee72959e-de1a-4721-a621-8ccdad306a01" containerName="revoke-keys"
Oct 30 13:29:11.011300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.010786    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee72959e-de1a-4721-a621-8ccdad306a01" containerName="revoke-keys"
Oct 30 13:29:11.011300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.010963    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee72959e-de1a-4721-a621-8ccdad306a01" containerName="revoke-keys"
Oct 30 13:29:11.017636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.017608    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.035606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.033998    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"]
Oct 30 13:29:11.058535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.058504    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:29:11.058875 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.058855    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:29:11.059040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.059026    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:29:11.059285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.059266    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:29:11.125246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.124955    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ts2w\" (UniqueName: \"kubernetes.io/projected/0888d296-9a88-4d16-860b-56bac84c654a-kube-api-access-7ts2w\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.125246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.125246    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.125538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.125284    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.226862 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.226821    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ts2w\" (UniqueName: \"kubernetes.io/projected/0888d296-9a88-4d16-860b-56bac84c654a-kube-api-access-7ts2w\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.227117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.227094    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.227196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.227135    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.227821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.227799    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.229576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.229547    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.250127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.250088    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ts2w\" (UniqueName: \"kubernetes.io/projected/0888d296-9a88-4d16-860b-56bac84c654a-kube-api-access-7ts2w\") pod \"maas-api-revoke-sub-101d25bb-64457\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.342264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.342148    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:11.648980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.648943    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"]
Oct 30 13:29:11.651529 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:11.651491    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0888d296_9a88_4d16_860b_56bac84c654a.slice/crio-f3cfdf29679f14b5665ade12cc0ce9e2d3ecde24bb900c24514fbc4d5238822e WatchSource:0}: Error finding container f3cfdf29679f14b5665ade12cc0ce9e2d3ecde24bb900c24514fbc4d5238822e: Status 404 returned error can't find the container with id f3cfdf29679f14b5665ade12cc0ce9e2d3ecde24bb900c24514fbc4d5238822e
Oct 30 13:29:11.873730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:11.873686    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457" event={"ID":"0888d296-9a88-4d16-860b-56bac84c654a","Type":"ContainerStarted","Data":"f3cfdf29679f14b5665ade12cc0ce9e2d3ecde24bb900c24514fbc4d5238822e"}
Oct 30 13:29:12.883610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:12.883569    4993 generic.go:356] "Generic (PLEG): container finished" podID="0888d296-9a88-4d16-860b-56bac84c654a" containerID="5846f3cc599959620a701bc26d69eed5e06fdbc396cfe1d0f8f92d1fa288a5e4" exitCode=0
Oct 30 13:29:12.884100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:12.883646    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457" event={"ID":"0888d296-9a88-4d16-860b-56bac84c654a","Type":"ContainerDied","Data":"5846f3cc599959620a701bc26d69eed5e06fdbc396cfe1d0f8f92d1fa288a5e4"}
Oct 30 13:29:14.151947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.151878    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:14.213539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.213507    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0888d296-9a88-4d16-860b-56bac84c654a-kube-api-access-7ts2w\" (UniqueName: \"kubernetes.io/projected/0888d296-9a88-4d16-860b-56bac84c654a-kube-api-access-7ts2w\") pod \"0888d296-9a88-4d16-860b-56bac84c654a\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") "
Oct 30 13:29:14.213753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.213562    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-openshift-ca\") pod \"0888d296-9a88-4d16-860b-56bac84c654a\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") "
Oct 30 13:29:14.213753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.213601    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-xks-ca\") pod \"0888d296-9a88-4d16-860b-56bac84c654a\" (UID: \"0888d296-9a88-4d16-860b-56bac84c654a\") "
Oct 30 13:29:14.214065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.214033    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-openshift-ca" pod "0888d296-9a88-4d16-860b-56bac84c654a" (UID: "0888d296-9a88-4d16-860b-56bac84c654a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:14.214292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.214267    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:14.216191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.216166    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-xks-ca" pod "0888d296-9a88-4d16-860b-56bac84c654a" (UID: "0888d296-9a88-4d16-860b-56bac84c654a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:14.216563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.216533    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0888d296-9a88-4d16-860b-56bac84c654a-kube-api-access-7ts2w" pod "0888d296-9a88-4d16-860b-56bac84c654a" (UID: "0888d296-9a88-4d16-860b-56bac84c654a"). InnerVolumeSpecName "kube-api-access-7ts2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:14.315961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.315919    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0888d296-9a88-4d16-860b-56bac84c654a-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:14.315961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.315959    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ts2w\" (UniqueName: \"kubernetes.io/projected/0888d296-9a88-4d16-860b-56bac84c654a-kube-api-access-7ts2w\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:14.800758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.800728    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:29:14.882507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.882473    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"]
Oct 30 13:29:14.912051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.912012    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"
Oct 30 13:29:14.912051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.912040    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457" event={"ID":"0888d296-9a88-4d16-860b-56bac84c654a","Type":"ContainerDied","Data":"f3cfdf29679f14b5665ade12cc0ce9e2d3ecde24bb900c24514fbc4d5238822e"}
Oct 30 13:29:14.912359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:14.912100    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3cfdf29679f14b5665ade12cc0ce9e2d3ecde24bb900c24514fbc4d5238822e"
Oct 30 13:29:15.568336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.568290    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf" podUID="69fc2dca-52be-445a-b1df-ca95a05605f8" containerName="payload-processing" containerID="cri-o://d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86" gracePeriod=25
Oct 30 13:29:15.909921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.909890    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:29:15.924495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.924452    4993 generic.go:356] "Generic (PLEG): container finished" podID="69fc2dca-52be-445a-b1df-ca95a05605f8" containerID="d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86" exitCode=0
Oct 30 13:29:15.924495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.924498    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"
Oct 30 13:29:15.924750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.924537    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf" event={"ID":"69fc2dca-52be-445a-b1df-ca95a05605f8","Type":"ContainerDied","Data":"d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86"}
Oct 30 13:29:15.924750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.924626    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf" event={"ID":"69fc2dca-52be-445a-b1df-ca95a05605f8","Type":"ContainerDied","Data":"50b03b6dedc01e79b3db42b2d20375c6a8d15a1750b0378947c96422335e5e6b"}
Oct 30 13:29:15.924750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.924651    4993 scope.go:122] "RemoveContainer" containerID="d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86"
Oct 30 13:29:15.935240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.935217    4993 scope.go:122] "RemoveContainer" containerID="d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86"
Oct 30 13:29:15.935574 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:29:15.935551    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86\": container with ID starting with d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86 not found: ID does not exist" containerID="d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86"
Oct 30 13:29:15.935633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.935584    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86"} err="failed to get container status \"d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86\": rpc error: code = NotFound desc = could not find container \"d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86\": container with ID starting with d917a22b87f7062fbc38b6debca6c2c4abb70a06448581c12deb5b78c35a1e86 not found: ID does not exist"
Oct 30 13:29:15.941950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.941926    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-plugins-config-volume\") pod \"69fc2dca-52be-445a-b1df-ca95a05605f8\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") "
Oct 30 13:29:15.942032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.941960    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-kube-api-access-j2wcj\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-kube-api-access-j2wcj\") pod \"69fc2dca-52be-445a-b1df-ca95a05605f8\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") "
Oct 30 13:29:15.942168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.942152    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-routing-overlay-volume\") pod \"69fc2dca-52be-445a-b1df-ca95a05605f8\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") "
Oct 30 13:29:15.942277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.942259    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-provider-credentials\") pod \"69fc2dca-52be-445a-b1df-ca95a05605f8\" (UID: \"69fc2dca-52be-445a-b1df-ca95a05605f8\") "
Oct 30 13:29:15.942497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.942363    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-plugins-config-volume" pod "69fc2dca-52be-445a-b1df-ca95a05605f8" (UID: "69fc2dca-52be-445a-b1df-ca95a05605f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:15.942594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.942524    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-routing-overlay-volume" pod "69fc2dca-52be-445a-b1df-ca95a05605f8" (UID: "69fc2dca-52be-445a-b1df-ca95a05605f8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:15.942901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.942882    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.942961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.942907    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc2dca-52be-445a-b1df-ca95a05605f8-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.944104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.944082    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-kube-api-access-j2wcj" pod "69fc2dca-52be-445a-b1df-ca95a05605f8" (UID: "69fc2dca-52be-445a-b1df-ca95a05605f8"). InnerVolumeSpecName "kube-api-access-j2wcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:15.944289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:15.944266    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-provider-credentials" pod "69fc2dca-52be-445a-b1df-ca95a05605f8" (UID: "69fc2dca-52be-445a-b1df-ca95a05605f8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:16.043982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:16.043947    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2wcj\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-kube-api-access-j2wcj\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:16.043982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:16.043980    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69fc2dca-52be-445a-b1df-ca95a05605f8-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:16.288840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:16.288798    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"]
Oct 30 13:29:16.302276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:16.302239    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-595d4xlrf"]
Oct 30 13:29:17.693157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:17.693115    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fc2dca-52be-445a-b1df-ca95a05605f8" path="/var/lib/kubelet/pods/69fc2dca-52be-445a-b1df-ca95a05605f8/volumes"
Oct 30 13:29:19.883959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:19.883388    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm" podUID="7780546f-5188-4cdf-b5dd-dc1156443f13" containerName="payload-processing" containerID="cri-o://e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db" gracePeriod=25
Oct 30 13:29:20.195701 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.195664    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:29:20.303435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.303393    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-kube-api-access-pktll\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-kube-api-access-pktll\") pod \"7780546f-5188-4cdf-b5dd-dc1156443f13\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") "
Oct 30 13:29:20.303644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.303628    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-routing-overlay-volume\") pod \"7780546f-5188-4cdf-b5dd-dc1156443f13\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") "
Oct 30 13:29:20.303691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.303674    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-plugins-config-volume\") pod \"7780546f-5188-4cdf-b5dd-dc1156443f13\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") "
Oct 30 13:29:20.304130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.303837    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-provider-credentials\") pod \"7780546f-5188-4cdf-b5dd-dc1156443f13\" (UID: \"7780546f-5188-4cdf-b5dd-dc1156443f13\") "
Oct 30 13:29:20.304130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.304053    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-routing-overlay-volume" pod "7780546f-5188-4cdf-b5dd-dc1156443f13" (UID: "7780546f-5188-4cdf-b5dd-dc1156443f13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:20.304130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.304099    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-plugins-config-volume" pod "7780546f-5188-4cdf-b5dd-dc1156443f13" (UID: "7780546f-5188-4cdf-b5dd-dc1156443f13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:20.304567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.304539    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:20.304707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.304570    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7780546f-5188-4cdf-b5dd-dc1156443f13-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:20.306032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.306006    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-provider-credentials" pod "7780546f-5188-4cdf-b5dd-dc1156443f13" (UID: "7780546f-5188-4cdf-b5dd-dc1156443f13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:20.306181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.306161    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-kube-api-access-pktll" pod "7780546f-5188-4cdf-b5dd-dc1156443f13" (UID: "7780546f-5188-4cdf-b5dd-dc1156443f13"). InnerVolumeSpecName "kube-api-access-pktll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:20.405905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.405811    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:20.405905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.405855    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktll\" (UniqueName: \"kubernetes.io/projected/7780546f-5188-4cdf-b5dd-dc1156443f13-kube-api-access-pktll\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:20.967915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.967876    4993 generic.go:356] "Generic (PLEG): container finished" podID="7780546f-5188-4cdf-b5dd-dc1156443f13" containerID="e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db" exitCode=0
Oct 30 13:29:20.968387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.967938    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"
Oct 30 13:29:20.968387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.967965    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm" event={"ID":"7780546f-5188-4cdf-b5dd-dc1156443f13","Type":"ContainerDied","Data":"e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db"}
Oct 30 13:29:20.968387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.968020    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm" event={"ID":"7780546f-5188-4cdf-b5dd-dc1156443f13","Type":"ContainerDied","Data":"7cc791edb911ef4164c11c792529164a5354f86f678e10abcf32a2afad8fcbdb"}
Oct 30 13:29:20.968387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.968048    4993 scope.go:122] "RemoveContainer" containerID="e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db"
Oct 30 13:29:20.979935 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.979916    4993 scope.go:122] "RemoveContainer" containerID="e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db"
Oct 30 13:29:20.980199 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:29:20.980178    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db\": container with ID starting with e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db not found: ID does not exist" containerID="e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db"
Oct 30 13:29:20.980299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:20.980234    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db"} err="failed to get container status \"e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db\": rpc error: code = NotFound desc = could not find container \"e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db\": container with ID starting with e651daafaafc8aba5ab50e9030aa5b21223023c857938a399d1f3f33754059db not found: ID does not exist"
Oct 30 13:29:21.023256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:21.023221    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"]
Oct 30 13:29:21.034833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:21.034803    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bb7rrlm"]
Oct 30 13:29:21.693756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:21.693720    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7780546f-5188-4cdf-b5dd-dc1156443f13" path="/var/lib/kubelet/pods/7780546f-5188-4cdf-b5dd-dc1156443f13/volumes"
Oct 30 13:29:22.614994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.614922    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"]
Oct 30 13:29:22.615973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.615917    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0888d296-9a88-4d16-860b-56bac84c654a" containerName="revoke-keys"
Oct 30 13:29:22.615973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.615945    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0888d296-9a88-4d16-860b-56bac84c654a" containerName="revoke-keys"
Oct 30 13:29:22.615973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.615969    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7780546f-5188-4cdf-b5dd-dc1156443f13" containerName="payload-processing"
Oct 30 13:29:22.616144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.615978    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7780546f-5188-4cdf-b5dd-dc1156443f13" containerName="payload-processing"
Oct 30 13:29:22.616144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.616016    4993 cpu_manager.go:414] "containerMap: removing container" podUID="69fc2dca-52be-445a-b1df-ca95a05605f8" containerName="payload-processing"
Oct 30 13:29:22.616144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.616024    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fc2dca-52be-445a-b1df-ca95a05605f8" containerName="payload-processing"
Oct 30 13:29:22.616307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.616177    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0888d296-9a88-4d16-860b-56bac84c654a" containerName="revoke-keys"
Oct 30 13:29:22.616307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.616193    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7780546f-5188-4cdf-b5dd-dc1156443f13" containerName="payload-processing"
Oct 30 13:29:22.616307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.616220    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fc2dca-52be-445a-b1df-ca95a05605f8" containerName="payload-processing"
Oct 30 13:29:22.621885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.621863    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.639486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.639457    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"]
Oct 30 13:29:22.645842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.645815    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0505b0\""
Oct 30 13:29:22.646012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.645816    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"routing-overlay\""
Oct 30 13:29:22.646012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.645815    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"agc-cred-fafbf0960658\""
Oct 30 13:29:22.646012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.645906    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"kube-root-ca.crt\""
Oct 30 13:29:22.649050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.649017    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-e2e--d83bf907-dockercfg-cxtkv\""
Oct 30 13:29:22.649050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.649027    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"openshift-service-ca.crt\""
Oct 30 13:29:22.738821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.738785    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.739036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.738948    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.739036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.739007    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.739197 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.739173    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdlw\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-kube-api-access-rtdlw\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.840715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.840683    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.840905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.840781    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.840905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.840814    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.840905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.840873    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdlw\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-kube-api-access-rtdlw\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.841493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.841468    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.841565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.841505    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.843398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.843379    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.864911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.864877    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdlw\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-kube-api-access-rtdlw\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:22.939298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:22.937457    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:23.295689 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:23.295647    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5503baa_6c92_473a_957a_c2adcbd09b82.slice/crio-6510ab3dd66249fae8984559fe1c9c88d024ff93dc3b372cca02415917b10d0e WatchSource:0}: Error finding container 6510ab3dd66249fae8984559fe1c9c88d024ff93dc3b372cca02415917b10d0e: Status 404 returned error can't find the container with id 6510ab3dd66249fae8984559fe1c9c88d024ff93dc3b372cca02415917b10d0e
Oct 30 13:29:23.305821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:23.305781    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"]
Oct 30 13:29:23.361249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:23.360977    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"]
Oct 30 13:29:24.007139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:24.007093    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km" event={"ID":"c5503baa-6c92-473a-957a-c2adcbd09b82","Type":"ContainerStarted","Data":"e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59"}
Oct 30 13:29:24.007654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:24.007150    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km" event={"ID":"c5503baa-6c92-473a-957a-c2adcbd09b82","Type":"ContainerStarted","Data":"6510ab3dd66249fae8984559fe1c9c88d024ff93dc3b372cca02415917b10d0e"}
Oct 30 13:29:24.007654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:24.007365    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:24.048773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:24.048712    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km" podStartSLOduration=2.048695582 podStartE2EDuration="2.048695582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:24.047994236 +0000 UTC m=+3428.523713846" watchObservedRunningTime="2026-10-30 13:29:24.048695582 +0000 UTC m=+3428.524415194"
Oct 30 13:29:27.817220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.817178    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"]
Oct 30 13:29:27.821602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.821579    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:27.836993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.836967    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"agc-cred-96e2063d6bd9\""
Oct 30 13:29:27.848670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.848638    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-63499e\""
Oct 30 13:29:27.848853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.848686    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"payload-processing-external-model-e2e--8c883ad3-dockercfg-fk89l\""
Oct 30 13:29:27.848853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.848685    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"routing-overlay\""
Oct 30 13:29:27.849147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.849128    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"openshift-service-ca.crt\""
Oct 30 13:29:27.849437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.849420    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-63499e\"/\"kube-root-ca.crt\""
Oct 30 13:29:27.856571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.856545    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"]
Oct 30 13:29:27.935710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.935656    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:27.935954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.935844    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:27.936044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.935991    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxwf\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-kube-api-access-dgxwf\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:27.936044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:27.936032    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.037009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.036965    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.037409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.037383    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxwf\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-kube-api-access-dgxwf\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.049263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.049175    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.056503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.049630    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.058669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.058642    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.059646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.059623    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.071138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.071060    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.102256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.099764    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxwf\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-kube-api-access-dgxwf\") pod \"payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") " pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.138267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.138230    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:28.548656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:28.548621    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"]
Oct 30 13:29:28.550413 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:28.550384    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa35769e_4296_4984_9033_478847869c19.slice/crio-1a1f604792304891189f2a695a01205140e9a6f71fe7f26de77f85ac68923877 WatchSource:0}: Error finding container 1a1f604792304891189f2a695a01205140e9a6f71fe7f26de77f85ac68923877: Status 404 returned error can't find the container with id 1a1f604792304891189f2a695a01205140e9a6f71fe7f26de77f85ac68923877
Oct 30 13:29:29.007584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.007538    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km" podUID="c5503baa-6c92-473a-957a-c2adcbd09b82" containerName="payload-processing" containerID="cri-o://e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59" gracePeriod=25
Oct 30 13:29:29.091860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.091821    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq" event={"ID":"aa35769e-4296-4984-9033-478847869c19","Type":"ContainerStarted","Data":"dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc"}
Oct 30 13:29:29.092081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.091889    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq" event={"ID":"aa35769e-4296-4984-9033-478847869c19","Type":"ContainerStarted","Data":"1a1f604792304891189f2a695a01205140e9a6f71fe7f26de77f85ac68923877"}
Oct 30 13:29:29.092081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.091959    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:29.137346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.137286    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq" podStartSLOduration=2.137270428 podStartE2EDuration="2.137270428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:29.132092409 +0000 UTC m=+3433.607812022" watchObservedRunningTime="2026-10-30 13:29:29.137270428 +0000 UTC m=+3433.612990038"
Oct 30 13:29:29.330960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.330931    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:29.481890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.481850    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-provider-credentials\") pod \"c5503baa-6c92-473a-957a-c2adcbd09b82\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") "
Oct 30 13:29:29.482110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.482093    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-routing-overlay-volume\") pod \"c5503baa-6c92-473a-957a-c2adcbd09b82\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") "
Oct 30 13:29:29.482165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.482141    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-kube-api-access-rtdlw\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-kube-api-access-rtdlw\") pod \"c5503baa-6c92-473a-957a-c2adcbd09b82\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") "
Oct 30 13:29:29.482270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.482254    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-plugins-config-volume\") pod \"c5503baa-6c92-473a-957a-c2adcbd09b82\" (UID: \"c5503baa-6c92-473a-957a-c2adcbd09b82\") "
Oct 30 13:29:29.482539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.482508    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-routing-overlay-volume" pod "c5503baa-6c92-473a-957a-c2adcbd09b82" (UID: "c5503baa-6c92-473a-957a-c2adcbd09b82"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:29.482699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.482677    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-plugins-config-volume" pod "c5503baa-6c92-473a-957a-c2adcbd09b82" (UID: "c5503baa-6c92-473a-957a-c2adcbd09b82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:29.482896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.482850    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.482896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.482874    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5503baa-6c92-473a-957a-c2adcbd09b82-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.484587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.484556    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-provider-credentials" pod "c5503baa-6c92-473a-957a-c2adcbd09b82" (UID: "c5503baa-6c92-473a-957a-c2adcbd09b82"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:29.484874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.484847    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-kube-api-access-rtdlw" pod "c5503baa-6c92-473a-957a-c2adcbd09b82" (UID: "c5503baa-6c92-473a-957a-c2adcbd09b82"). InnerVolumeSpecName "kube-api-access-rtdlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:29.584337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.584244    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.584337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:29.584287    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtdlw\" (UniqueName: \"kubernetes.io/projected/c5503baa-6c92-473a-957a-c2adcbd09b82-kube-api-access-rtdlw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:30.103798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.103757    4993 generic.go:356] "Generic (PLEG): container finished" podID="c5503baa-6c92-473a-957a-c2adcbd09b82" containerID="e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59" exitCode=0
Oct 30 13:29:30.104355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.103810    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"
Oct 30 13:29:30.104355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.103838    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km" event={"ID":"c5503baa-6c92-473a-957a-c2adcbd09b82","Type":"ContainerDied","Data":"e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59"}
Oct 30 13:29:30.104355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.103898    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km" event={"ID":"c5503baa-6c92-473a-957a-c2adcbd09b82","Type":"ContainerDied","Data":"6510ab3dd66249fae8984559fe1c9c88d024ff93dc3b372cca02415917b10d0e"}
Oct 30 13:29:30.104355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.103928    4993 scope.go:122] "RemoveContainer" containerID="e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59"
Oct 30 13:29:30.115501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.115471    4993 scope.go:122] "RemoveContainer" containerID="e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59"
Oct 30 13:29:30.115863 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:29:30.115838    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59\": container with ID starting with e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59 not found: ID does not exist" containerID="e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59"
Oct 30 13:29:30.115944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.115878    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59"} err="failed to get container status \"e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59\": rpc error: code = NotFound desc = could not find container \"e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59\": container with ID starting with e4d04e0901b10afe49831acbba2e7866b5ea02c214195bc7074f574b0ad1bf59 not found: ID does not exist"
Oct 30 13:29:30.210244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.210183    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"]
Oct 30 13:29:30.229664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:30.229623    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579c4t2km"]
Oct 30 13:29:31.709231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:31.709095    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5503baa-6c92-473a-957a-c2adcbd09b82" path="/var/lib/kubelet/pods/c5503baa-6c92-473a-957a-c2adcbd09b82/volumes"
Oct 30 13:29:35.940895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.940855    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"]
Oct 30 13:29:35.942956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.941464    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c5503baa-6c92-473a-957a-c2adcbd09b82" containerName="payload-processing"
Oct 30 13:29:35.942956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.941476    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5503baa-6c92-473a-957a-c2adcbd09b82" containerName="payload-processing"
Oct 30 13:29:35.942956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.941585    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5503baa-6c92-473a-957a-c2adcbd09b82" containerName="payload-processing"
Oct 30 13:29:35.944881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.944864    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:35.966799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.966763    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"]
Oct 30 13:29:35.972891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.971553    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:29:35.972891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.971635    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:29:35.972891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.971661    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:29:35.972891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:35.971721    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:29:36.009909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.009866    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.010129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.009937    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjk4j\" (UniqueName: \"kubernetes.io/projected/1413ba77-60ff-4ce2-978a-250a635b7817-kube-api-access-fjk4j\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.010129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.010050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.112122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.112079    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.112122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.112131    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjk4j\" (UniqueName: \"kubernetes.io/projected/1413ba77-60ff-4ce2-978a-250a635b7817-kube-api-access-fjk4j\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.112444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.112152    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.112813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.112780    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.114626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.114599    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.136858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.136819    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjk4j\" (UniqueName: \"kubernetes.io/projected/1413ba77-60ff-4ce2-978a-250a635b7817-kube-api-access-fjk4j\") pod \"maas-api-revoke-sub-3721a947-dxdqz\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.259656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.259608    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:36.320717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.319632    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"]
Oct 30 13:29:36.326247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.326193    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.354973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.354938    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"]
Oct 30 13:29:36.416867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.416817    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p2ll\" (UniqueName: \"kubernetes.io/projected/20ceaba3-f33a-40b2-a76d-abafba657a14-kube-api-access-2p2ll\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.417088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.417025    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.417198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.417170    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.518248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.518147    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.518415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.518269    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2ll\" (UniqueName: \"kubernetes.io/projected/20ceaba3-f33a-40b2-a76d-abafba657a14-kube-api-access-2p2ll\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.518415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.518380    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.518811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.518787    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.520937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.520911    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.541532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.541496    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2ll\" (UniqueName: \"kubernetes.io/projected/20ceaba3-f33a-40b2-a76d-abafba657a14-kube-api-access-2p2ll\") pod \"maas-api-revoke-sub-b7d94d4e-pv42t\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.572303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.572274    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"]
Oct 30 13:29:36.573963 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:36.573925    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1413ba77_60ff_4ce2_978a_250a635b7817.slice/crio-6374e3e813b7db86ac4ff3cf34d700234b41b650d5bb2c9204d7496987ab910a WatchSource:0}: Error finding container 6374e3e813b7db86ac4ff3cf34d700234b41b650d5bb2c9204d7496987ab910a: Status 404 returned error can't find the container with id 6374e3e813b7db86ac4ff3cf34d700234b41b650d5bb2c9204d7496987ab910a
Oct 30 13:29:36.661468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.661440    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:36.969099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:36.969066    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"]
Oct 30 13:29:36.970828 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:36.970798    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ceaba3_f33a_40b2_a76d_abafba657a14.slice/crio-089b57e2e2be6fbb796a4f04cfebd6831256d90dcd43f27298908b5dea5d5160 WatchSource:0}: Error finding container 089b57e2e2be6fbb796a4f04cfebd6831256d90dcd43f27298908b5dea5d5160: Status 404 returned error can't find the container with id 089b57e2e2be6fbb796a4f04cfebd6831256d90dcd43f27298908b5dea5d5160
Oct 30 13:29:37.245961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:37.245912    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz" event={"ID":"1413ba77-60ff-4ce2-978a-250a635b7817","Type":"ContainerStarted","Data":"948138374f3b6253e08a5f3af143209a5c1fd76fba4721260d494417422d197c"}
Oct 30 13:29:37.246173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:37.245987    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz" event={"ID":"1413ba77-60ff-4ce2-978a-250a635b7817","Type":"ContainerStarted","Data":"6374e3e813b7db86ac4ff3cf34d700234b41b650d5bb2c9204d7496987ab910a"}
Oct 30 13:29:37.247492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:37.247465    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t" event={"ID":"20ceaba3-f33a-40b2-a76d-abafba657a14","Type":"ContainerStarted","Data":"089b57e2e2be6fbb796a4f04cfebd6831256d90dcd43f27298908b5dea5d5160"}
Oct 30 13:29:37.297870 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:37.297816    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz" podStartSLOduration=2.29779998 podStartE2EDuration="2.29779998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:37.294155732 +0000 UTC m=+3441.769875342" watchObservedRunningTime="2026-10-30 13:29:37.29779998 +0000 UTC m=+3441.773519599"
Oct 30 13:29:38.257603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:38.257570    4993 generic.go:356] "Generic (PLEG): container finished" podID="20ceaba3-f33a-40b2-a76d-abafba657a14" containerID="a4f5959007f0a1ff0b246c1a857fc195ddd5aa0b935a6a155980eb736170b7f5" exitCode=0
Oct 30 13:29:38.258076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:38.257654    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t" event={"ID":"20ceaba3-f33a-40b2-a76d-abafba657a14","Type":"ContainerDied","Data":"a4f5959007f0a1ff0b246c1a857fc195ddd5aa0b935a6a155980eb736170b7f5"}
Oct 30 13:29:38.259464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:38.259437    4993 generic.go:356] "Generic (PLEG): container finished" podID="1413ba77-60ff-4ce2-978a-250a635b7817" containerID="948138374f3b6253e08a5f3af143209a5c1fd76fba4721260d494417422d197c" exitCode=0
Oct 30 13:29:38.259586 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:38.259483    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz" event={"ID":"1413ba77-60ff-4ce2-978a-250a635b7817","Type":"ContainerDied","Data":"948138374f3b6253e08a5f3af143209a5c1fd76fba4721260d494417422d197c"}
Oct 30 13:29:38.569848 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:29:38.569743    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:29:39.960083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:39.960054    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:39.964957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:39.964933    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:40.097367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.097276    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-openshift-ca\") pod \"20ceaba3-f33a-40b2-a76d-abafba657a14\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") "
Oct 30 13:29:40.097367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.097344    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-openshift-ca\") pod \"1413ba77-60ff-4ce2-978a-250a635b7817\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") "
Oct 30 13:29:40.097617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.097399    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1413ba77-60ff-4ce2-978a-250a635b7817-kube-api-access-fjk4j\" (UniqueName: \"kubernetes.io/projected/1413ba77-60ff-4ce2-978a-250a635b7817-kube-api-access-fjk4j\") pod \"1413ba77-60ff-4ce2-978a-250a635b7817\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") "
Oct 30 13:29:40.097617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.097476    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ceaba3-f33a-40b2-a76d-abafba657a14-kube-api-access-2p2ll\" (UniqueName: \"kubernetes.io/projected/20ceaba3-f33a-40b2-a76d-abafba657a14-kube-api-access-2p2ll\") pod \"20ceaba3-f33a-40b2-a76d-abafba657a14\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") "
Oct 30 13:29:40.097736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.097658    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-xks-ca\") pod \"20ceaba3-f33a-40b2-a76d-abafba657a14\" (UID: \"20ceaba3-f33a-40b2-a76d-abafba657a14\") "
Oct 30 13:29:40.097794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.097756    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-xks-ca\") pod \"1413ba77-60ff-4ce2-978a-250a635b7817\" (UID: \"1413ba77-60ff-4ce2-978a-250a635b7817\") "
Oct 30 13:29:40.099332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.099298    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-openshift-ca" pod "20ceaba3-f33a-40b2-a76d-abafba657a14" (UID: "20ceaba3-f33a-40b2-a76d-abafba657a14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:40.099490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.099363    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-openshift-ca" pod "1413ba77-60ff-4ce2-978a-250a635b7817" (UID: "1413ba77-60ff-4ce2-978a-250a635b7817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:40.102644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.102609    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-xks-ca" pod "20ceaba3-f33a-40b2-a76d-abafba657a14" (UID: "20ceaba3-f33a-40b2-a76d-abafba657a14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:40.104514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.104329    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-xks-ca" pod "1413ba77-60ff-4ce2-978a-250a635b7817" (UID: "1413ba77-60ff-4ce2-978a-250a635b7817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:40.105345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.105319    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1413ba77-60ff-4ce2-978a-250a635b7817-kube-api-access-fjk4j" pod "1413ba77-60ff-4ce2-978a-250a635b7817" (UID: "1413ba77-60ff-4ce2-978a-250a635b7817"). InnerVolumeSpecName "kube-api-access-fjk4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:40.108091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.108047    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ceaba3-f33a-40b2-a76d-abafba657a14-kube-api-access-2p2ll" pod "20ceaba3-f33a-40b2-a76d-abafba657a14" (UID: "20ceaba3-f33a-40b2-a76d-abafba657a14"). InnerVolumeSpecName "kube-api-access-2p2ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:40.108832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.108814    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:29:40.200107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.200066    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p2ll\" (UniqueName: \"kubernetes.io/projected/20ceaba3-f33a-40b2-a76d-abafba657a14-kube-api-access-2p2ll\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:40.200366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.200346    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:40.200551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.200536    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:40.200651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.200629    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ceaba3-f33a-40b2-a76d-abafba657a14-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:40.200651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.200652    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1413ba77-60ff-4ce2-978a-250a635b7817-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:40.200817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.200666    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjk4j\" (UniqueName: \"kubernetes.io/projected/1413ba77-60ff-4ce2-978a-250a635b7817-kube-api-access-fjk4j\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:40.282900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.282870    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"
Oct 30 13:29:40.283198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.282905    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz" event={"ID":"1413ba77-60ff-4ce2-978a-250a635b7817","Type":"ContainerDied","Data":"6374e3e813b7db86ac4ff3cf34d700234b41b650d5bb2c9204d7496987ab910a"}
Oct 30 13:29:40.283198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.282995    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6374e3e813b7db86ac4ff3cf34d700234b41b650d5bb2c9204d7496987ab910a"
Oct 30 13:29:40.284782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.284758    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"
Oct 30 13:29:40.284934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.284775    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t" event={"ID":"20ceaba3-f33a-40b2-a76d-abafba657a14","Type":"ContainerDied","Data":"089b57e2e2be6fbb796a4f04cfebd6831256d90dcd43f27298908b5dea5d5160"}
Oct 30 13:29:40.284934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.284835    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="089b57e2e2be6fbb796a4f04cfebd6831256d90dcd43f27298908b5dea5d5160"
Oct 30 13:29:40.319732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.319689    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"]
Oct 30 13:29:40.320493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.320470    4993 cpu_manager.go:414] "containerMap: removing container" podUID="20ceaba3-f33a-40b2-a76d-abafba657a14" containerName="revoke-keys"
Oct 30 13:29:40.320493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.320493    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ceaba3-f33a-40b2-a76d-abafba657a14" containerName="revoke-keys"
Oct 30 13:29:40.320686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.320527    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1413ba77-60ff-4ce2-978a-250a635b7817" containerName="revoke-keys"
Oct 30 13:29:40.320686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.320534    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1413ba77-60ff-4ce2-978a-250a635b7817" containerName="revoke-keys"
Oct 30 13:29:40.320805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.320722    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ceaba3-f33a-40b2-a76d-abafba657a14" containerName="revoke-keys"
Oct 30 13:29:40.320805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.320739    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1413ba77-60ff-4ce2-978a-250a635b7817" containerName="revoke-keys"
Oct 30 13:29:40.324897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.324872    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.345943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.344147    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 13:29:40.345943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.344182    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 13:29:40.345943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.344378    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 13:29:40.345943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.344407    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3c6ac478ec12\""
Oct 30 13:29:40.345943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.344446    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8f9adf8757d7\""
Oct 30 13:29:40.345943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.344543    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 13:29:40.345943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.344581    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mfkdm\""
Oct 30 13:29:40.355596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.355520    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"]
Oct 30 13:29:40.404258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.404191    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-routing-overlay-volume\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.404441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.404397    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-kube-api-access-88mcq\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.404569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.404544    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-plugins-config-volume\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.404701 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.404686    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-provider-credentials\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.506323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.506267    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-plugins-config-volume\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.506551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.506446    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-provider-credentials\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.506551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.506486    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-routing-overlay-volume\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.506672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.506556    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-kube-api-access-88mcq\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.507116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.507093    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-plugins-config-volume\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.507731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.507707    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-routing-overlay-volume\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.509687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.509659    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-provider-credentials\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.530807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.530772    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-kube-api-access-88mcq\") pod \"payload-processing-external-model-6fdcbd645b-d9lrk\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") " pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.641468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.641369    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:40.989181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:40.988850    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"]
Oct 30 13:29:40.991394 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:40.991361    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4113d2_85d4_4849_9023_9721f00762e8.slice/crio-bbf115d7472fa7467dae64b372eca6308063fd2ea74c9196052a8cf55e2e32ec WatchSource:0}: Error finding container bbf115d7472fa7467dae64b372eca6308063fd2ea74c9196052a8cf55e2e32ec: Status 404 returned error can't find the container with id bbf115d7472fa7467dae64b372eca6308063fd2ea74c9196052a8cf55e2e32ec
Oct 30 13:29:41.295650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:41.295555    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk" event={"ID":"ea4113d2-85d4-4849-9023-9721f00762e8","Type":"ContainerStarted","Data":"481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881"}
Oct 30 13:29:41.295650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:41.295626    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:41.295650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:41.295646    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk" event={"ID":"ea4113d2-85d4-4849-9023-9721f00762e8","Type":"ContainerStarted","Data":"bbf115d7472fa7467dae64b372eca6308063fd2ea74c9196052a8cf55e2e32ec"}
Oct 30 13:29:41.340326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:41.340268    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk" podStartSLOduration=1.340250551 podStartE2EDuration="1.340250551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:41.337395848 +0000 UTC m=+3445.813115457" watchObservedRunningTime="2026-10-30 13:29:41.340250551 +0000 UTC m=+3445.815970161"
Oct 30 13:29:45.033649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:45.033611    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"]
Oct 30 13:29:45.044940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:45.044905    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13d7e35-mwddm"]
Oct 30 13:29:45.696310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:45.696263    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b1835a-0147-4811-b67d-e24f005665e4" path="/var/lib/kubelet/pods/44b1835a-0147-4811-b67d-e24f005665e4/volumes"
Oct 30 13:29:45.810044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:45.810003    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"]
Oct 30 13:29:48.948445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:48.948402    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"]
Oct 30 13:29:48.953353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:48.953318    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:48.977981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:48.977945    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"]
Oct 30 13:29:48.982512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:48.982476    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:29:48.982721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:48.982475    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:29:48.982887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:48.982519    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:29:48.983282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:48.983257    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:29:49.050726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.050681    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwh9\" (UniqueName: \"kubernetes.io/projected/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-kube-api-access-xqwh9\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.050972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.050936    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.051257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.051176    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.152488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.152448    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.152679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.152642    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.152744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.152687    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwh9\" (UniqueName: \"kubernetes.io/projected/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-kube-api-access-xqwh9\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.153463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.153429    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.155716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.155685    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.177643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.177608    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwh9\" (UniqueName: \"kubernetes.io/projected/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-kube-api-access-xqwh9\") pod \"maas-api-revoke-sub-fc055a43-lrxxh\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.274553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.274510    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:49.981711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:49.981668    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"]
Oct 30 13:29:49.982648 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:49.982615    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78cf351_9e8a_48ac_b0a0_2fdf44ab3363.slice/crio-c9c3a786ad6dac87ed6068b3e2f7c6157878bad73c217246a47a450f4edd4b6d WatchSource:0}: Error finding container c9c3a786ad6dac87ed6068b3e2f7c6157878bad73c217246a47a450f4edd4b6d: Status 404 returned error can't find the container with id c9c3a786ad6dac87ed6068b3e2f7c6157878bad73c217246a47a450f4edd4b6d
Oct 30 13:29:50.546278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:50.543156    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh" event={"ID":"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363","Type":"ContainerStarted","Data":"c7670b7069e86c280c548c2959c31ace4b1944b419d99cc29334461071a17348"}
Oct 30 13:29:50.546278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:50.543252    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh" event={"ID":"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363","Type":"ContainerStarted","Data":"c9c3a786ad6dac87ed6068b3e2f7c6157878bad73c217246a47a450f4edd4b6d"}
Oct 30 13:29:50.814510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:50.814285    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk" podUID="ea4113d2-85d4-4849-9023-9721f00762e8" containerName="payload-processing" containerID="cri-o://481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881" gracePeriod=25
Oct 30 13:29:51.174099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.174066    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:51.211123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.211081    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-routing-overlay-volume\") pod \"ea4113d2-85d4-4849-9023-9721f00762e8\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") "
Oct 30 13:29:51.211346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.211183    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-plugins-config-volume\") pod \"ea4113d2-85d4-4849-9023-9721f00762e8\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") "
Oct 30 13:29:51.211443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.211427    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-kube-api-access-88mcq\") pod \"ea4113d2-85d4-4849-9023-9721f00762e8\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") "
Oct 30 13:29:51.211566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.211538    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-provider-credentials\") pod \"ea4113d2-85d4-4849-9023-9721f00762e8\" (UID: \"ea4113d2-85d4-4849-9023-9721f00762e8\") "
Oct 30 13:29:51.211664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.211627    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-routing-overlay-volume" pod "ea4113d2-85d4-4849-9023-9721f00762e8" (UID: "ea4113d2-85d4-4849-9023-9721f00762e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:51.211729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.211696    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-plugins-config-volume" pod "ea4113d2-85d4-4849-9023-9721f00762e8" (UID: "ea4113d2-85d4-4849-9023-9721f00762e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:51.212089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.212061    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:51.212231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.212092    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4113d2-85d4-4849-9023-9721f00762e8-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:51.213974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.213949    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-kube-api-access-88mcq" pod "ea4113d2-85d4-4849-9023-9721f00762e8" (UID: "ea4113d2-85d4-4849-9023-9721f00762e8"). InnerVolumeSpecName "kube-api-access-88mcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:51.214394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.214375    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-provider-credentials" pod "ea4113d2-85d4-4849-9023-9721f00762e8" (UID: "ea4113d2-85d4-4849-9023-9721f00762e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:51.225132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.225064    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh" podStartSLOduration=3.225043114 podStartE2EDuration="3.225043114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:50.600729705 +0000 UTC m=+3455.076449316" watchObservedRunningTime="2026-10-30 13:29:51.225043114 +0000 UTC m=+3455.700762726"
Oct 30 13:29:51.313428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.313390    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:51.313428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.313423    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/ea4113d2-85d4-4849-9023-9721f00762e8-kube-api-access-88mcq\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:51.553023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.552983    4993 generic.go:356] "Generic (PLEG): container finished" podID="b78cf351-9e8a-48ac-b0a0-2fdf44ab3363" containerID="c7670b7069e86c280c548c2959c31ace4b1944b419d99cc29334461071a17348" exitCode=0
Oct 30 13:29:51.553261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.553078    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh" event={"ID":"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363","Type":"ContainerDied","Data":"c7670b7069e86c280c548c2959c31ace4b1944b419d99cc29334461071a17348"}
Oct 30 13:29:51.554997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.554971    4993 generic.go:356] "Generic (PLEG): container finished" podID="ea4113d2-85d4-4849-9023-9721f00762e8" containerID="481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881" exitCode=0
Oct 30 13:29:51.555137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.555026    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"
Oct 30 13:29:51.555137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.555062    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk" event={"ID":"ea4113d2-85d4-4849-9023-9721f00762e8","Type":"ContainerDied","Data":"481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881"}
Oct 30 13:29:51.555137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.555124    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk" event={"ID":"ea4113d2-85d4-4849-9023-9721f00762e8","Type":"ContainerDied","Data":"bbf115d7472fa7467dae64b372eca6308063fd2ea74c9196052a8cf55e2e32ec"}
Oct 30 13:29:51.555269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.555159    4993 scope.go:122] "RemoveContainer" containerID="481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881"
Oct 30 13:29:51.568657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.568606    4993 scope.go:122] "RemoveContainer" containerID="481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881"
Oct 30 13:29:51.569093 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:29:51.569067    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881\": container with ID starting with 481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881 not found: ID does not exist" containerID="481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881"
Oct 30 13:29:51.569179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.569107    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881"} err="failed to get container status \"481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881\": rpc error: code = NotFound desc = could not find container \"481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881\": container with ID starting with 481e3557697d014a49d3d04e53b68ee6a62edf531fcf90ce086ea4b70ca3d881 not found: ID does not exist"
Oct 30 13:29:51.588681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.588644    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"]
Oct 30 13:29:51.589308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.589287    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ea4113d2-85d4-4849-9023-9721f00762e8" containerName="payload-processing"
Oct 30 13:29:51.589308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.589308    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4113d2-85d4-4849-9023-9721f00762e8" containerName="payload-processing"
Oct 30 13:29:51.589453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.589438    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4113d2-85d4-4849-9023-9721f00762e8" containerName="payload-processing"
Oct 30 13:29:51.594431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.594405    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.619643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.619584    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-e2e--d83bf907-dockercfg-bqct5\""
Oct 30 13:29:51.620044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.620021    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0505b0\""
Oct 30 13:29:51.679481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.620384    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"agc-cred-61d98a35cf62\""
Oct 30 13:29:51.679794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.620469    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"kube-root-ca.crt\""
Oct 30 13:29:51.679948 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.620624    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"openshift-service-ca.crt\""
Oct 30 13:29:51.680089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.648364    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"routing-overlay\""
Oct 30 13:29:51.680326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.649372    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrjb\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-kube-api-access-5hrjb\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.680590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.679395    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"]
Oct 30 13:29:51.680835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.680817    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.681269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.681249    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.681517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.681500    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.762177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.762117    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"]
Oct 30 13:29:51.768512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.768483    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fdcbd645b-d9lrk"]
Oct 30 13:29:51.786395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.783613    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrjb\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-kube-api-access-5hrjb\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.786395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.783684    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.786395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.783755    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.786395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.783788    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.788262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.787106    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.788262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.787873    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.825992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.825875    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.830155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.828895    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrjb\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-kube-api-access-5hrjb\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:51.909821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:51.909782    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:52.218658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.218622    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"]
Oct 30 13:29:52.220107 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:52.220078    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25ae132_ab62_48e7_86e5_1c4517e51365.slice/crio-81e199ccbb7056c6b0cdc5dd4928337d31b74eb3d09afbe37a59c2638d209fd8 WatchSource:0}: Error finding container 81e199ccbb7056c6b0cdc5dd4928337d31b74eb3d09afbe37a59c2638d209fd8: Status 404 returned error can't find the container with id 81e199ccbb7056c6b0cdc5dd4928337d31b74eb3d09afbe37a59c2638d209fd8
Oct 30 13:29:52.572230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.571750    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2" event={"ID":"e25ae132-ab62-48e7-86e5-1c4517e51365","Type":"ContainerStarted","Data":"6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2"}
Oct 30 13:29:52.572230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.571901    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:29:52.572230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.571932    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2" event={"ID":"e25ae132-ab62-48e7-86e5-1c4517e51365","Type":"ContainerStarted","Data":"81e199ccbb7056c6b0cdc5dd4928337d31b74eb3d09afbe37a59c2638d209fd8"}
Oct 30 13:29:52.615630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.615568    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2" podStartSLOduration=1.6155528129999999 podStartE2EDuration="1.615552813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:52.613397929 +0000 UTC m=+3457.089117543" watchObservedRunningTime="2026-10-30 13:29:52.615552813 +0000 UTC m=+3457.091272423"
Oct 30 13:29:52.775115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.775082    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:52.904015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.903930    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-kube-api-access-xqwh9\" (UniqueName: \"kubernetes.io/projected/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-kube-api-access-xqwh9\") pod \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") "
Oct 30 13:29:52.904193 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.904075    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-openshift-ca\") pod \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") "
Oct 30 13:29:52.904350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.904328    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-xks-ca\") pod \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\" (UID: \"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363\") "
Oct 30 13:29:52.904522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.904499    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-openshift-ca" pod "b78cf351-9e8a-48ac-b0a0-2fdf44ab3363" (UID: "b78cf351-9e8a-48ac-b0a0-2fdf44ab3363"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:52.904821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.904802    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:52.906259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.906234    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-kube-api-access-xqwh9" pod "b78cf351-9e8a-48ac-b0a0-2fdf44ab3363" (UID: "b78cf351-9e8a-48ac-b0a0-2fdf44ab3363"). InnerVolumeSpecName "kube-api-access-xqwh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:52.906443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:52.906421    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-xks-ca" pod "b78cf351-9e8a-48ac-b0a0-2fdf44ab3363" (UID: "b78cf351-9e8a-48ac-b0a0-2fdf44ab3363"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:53.005751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:53.005715    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwh9\" (UniqueName: \"kubernetes.io/projected/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-kube-api-access-xqwh9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:53.005751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:53.005748    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:29:53.585242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:53.581186    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"
Oct 30 13:29:53.585242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:53.581182    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh" event={"ID":"b78cf351-9e8a-48ac-b0a0-2fdf44ab3363","Type":"ContainerDied","Data":"c9c3a786ad6dac87ed6068b3e2f7c6157878bad73c217246a47a450f4edd4b6d"}
Oct 30 13:29:53.585242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:53.581377    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c3a786ad6dac87ed6068b3e2f7c6157878bad73c217246a47a450f4edd4b6d"
Oct 30 13:29:53.695132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:53.695096    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4113d2-85d4-4849-9023-9721f00762e8" path="/var/lib/kubelet/pods/ea4113d2-85d4-4849-9023-9721f00762e8/volumes"
Oct 30 13:29:58.107236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.107161    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"]
Oct 30 13:29:58.108223 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.108185    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b78cf351-9e8a-48ac-b0a0-2fdf44ab3363" containerName="revoke-keys"
Oct 30 13:29:58.108367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.108229    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78cf351-9e8a-48ac-b0a0-2fdf44ab3363" containerName="revoke-keys"
Oct 30 13:29:58.108460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.108441    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78cf351-9e8a-48ac-b0a0-2fdf44ab3363" containerName="revoke-keys"
Oct 30 13:29:58.113148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.113122    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"]
Oct 30 13:29:58.113331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.113293    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.191880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.191837    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.192065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.191909    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.192065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.192032    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.192143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.192065    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.192143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.192098    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.192294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.192262    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.192336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.192312    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.192420 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.192397    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tfc\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-kube-api-access-78tfc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.192565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.192444    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293317    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293371    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293389    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293433    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tfc\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-kube-api-access-78tfc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293477    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293625    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293694    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293748    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.293865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.293781    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.294471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.294263    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.294471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.294388    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.294471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.294416    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.294673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.294635    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.294769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.294746    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.296724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.296693    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.297312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.297285    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.317621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.317587    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.318250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.318221    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tfc\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-kube-api-access-78tfc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-k879q\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.453361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.453272    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:29:58.780010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.779979    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"]
Oct 30 13:29:58.783068 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:29:58.783036    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3dfbab_21c7_43e3_a95b_197c84e685c7.slice/crio-8cb7fce55148052a1e42b61246f50180a610630edf2d57a3701c14912e2c0f6c WatchSource:0}: Error finding container 8cb7fce55148052a1e42b61246f50180a610630edf2d57a3701c14912e2c0f6c: Status 404 returned error can't find the container with id 8cb7fce55148052a1e42b61246f50180a610630edf2d57a3701c14912e2c0f6c
Oct 30 13:29:58.794240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.792110    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:29:58.794240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.792283    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:29:58.794240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:58.792373    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:29:59.712071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.712033    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"]
Oct 30 13:29:59.717191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.717163    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.719990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.719957    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"]
Oct 30 13:29:59.737001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.735958    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:29:59.747432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.747394    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:29:59.747625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.747464    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:29:59.747737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.747477    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:29:59.757283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.756317    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" event={"ID":"6f3dfbab-21c7-43e3-a95b-197c84e685c7","Type":"ContainerStarted","Data":"b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2"}
Oct 30 13:29:59.757283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.756390    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" event={"ID":"6f3dfbab-21c7-43e3-a95b-197c84e685c7","Type":"ContainerStarted","Data":"8cb7fce55148052a1e42b61246f50180a610630edf2d57a3701c14912e2c0f6c"}
Oct 30 13:29:59.829151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.827474    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46q4\" (UniqueName: \"kubernetes.io/projected/986a714d-458c-47dc-b878-79c74ffa9979-kube-api-access-v46q4\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.829151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.827692    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.829151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.827777    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.936754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.932693    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.936754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.932798    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.936754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.932985    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46q4\" (UniqueName: \"kubernetes.io/projected/986a714d-458c-47dc-b878-79c74ffa9979-kube-api-access-v46q4\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.936754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.934280    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.936754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.936691    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:29:59.946901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.946838    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podStartSLOduration=2.94681715 podStartE2EDuration="2.94681715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:59.940296249 +0000 UTC m=+3464.416015852" watchObservedRunningTime="2026-10-30 13:29:59.94681715 +0000 UTC m=+3464.422536761"
Oct 30 13:29:59.973116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:29:59.973031    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46q4\" (UniqueName: \"kubernetes.io/projected/986a714d-458c-47dc-b878-79c74ffa9979-kube-api-access-v46q4\") pod \"maas-api-revoke-sub-0f84fc84-s5t9z\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:30:00.039430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.039386    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:30:00.229871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.226150    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"]
Oct 30 13:30:00.326718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.326653    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"]
Oct 30 13:30:00.332646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.332605    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:00.338146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.338119    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"]
Oct 30 13:30:00.339553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.339389    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:00.380969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.380938    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"]
Oct 30 13:30:00.410897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.391845    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:00.463132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.445101    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6\" (UID: \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.497455    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"]
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.506477    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rch\" (UniqueName: \"kubernetes.io/projected/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-kube-api-access-98rch\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6\" (UID: \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.506580    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf87f\" (UniqueName: \"kubernetes.io/projected/5af816ef-1abb-4490-a7e0-4d1901c27223-kube-api-access-xf87f\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw\" (UID: \"5af816ef-1abb-4490-a7e0-4d1901c27223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.506641    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af816ef-1abb-4490-a7e0-4d1901c27223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw\" (UID: \"5af816ef-1abb-4490-a7e0-4d1901c27223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.506672    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6\" (UID: \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.506731    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972qg\" (UniqueName: \"kubernetes.io/projected/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-kube-api-access-972qg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6\" (UID: \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.506957    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.507010    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 13:30:00.532179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.507711    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:00.536846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.536817    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:30:00.539407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.539381    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"]
Oct 30 13:30:00.539593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.539579    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"]
Oct 30 13:30:00.539724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.539713    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"]
Oct 30 13:30:00.542296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.542062    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:00.558230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.545358    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"]
Oct 30 13:30:00.579275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.558534    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"]
Oct 30 13:30:00.616056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.547135    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:00.616056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.609142    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrzkf\" (UniqueName: \"kubernetes.io/projected/78a20603-e044-4279-a9e8-7c2c7d030a44-kube-api-access-vrzkf\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7\" (UID: \"78a20603-e044-4279-a9e8-7c2c7d030a44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:00.618617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.618582    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rch\" (UniqueName: \"kubernetes.io/projected/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-kube-api-access-98rch\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6\" (UID: \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:00.618907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.618886    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf87f\" (UniqueName: \"kubernetes.io/projected/5af816ef-1abb-4490-a7e0-4d1901c27223-kube-api-access-xf87f\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw\" (UID: \"5af816ef-1abb-4490-a7e0-4d1901c27223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:00.619065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.619048    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af816ef-1abb-4490-a7e0-4d1901c27223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw\" (UID: \"5af816ef-1abb-4490-a7e0-4d1901c27223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:00.619217 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.619185    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6\" (UID: \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:00.619392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.619374    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972qg\" (UniqueName: \"kubernetes.io/projected/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-kube-api-access-972qg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6\" (UID: \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:00.619645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.619624    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6\" (UID: \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:00.619867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.619846    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a20603-e044-4279-a9e8-7c2c7d030a44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7\" (UID: \"78a20603-e044-4279-a9e8-7c2c7d030a44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:00.620154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.619988    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6\" (UID: \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:00.620154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.618692    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:00.620154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.619889    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af816ef-1abb-4490-a7e0-4d1901c27223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw\" (UID: \"5af816ef-1abb-4490-a7e0-4d1901c27223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:00.620154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.618609    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"]
Oct 30 13:30:00.622160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.622134    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6\" (UID: \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:00.628629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.628602    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"]
Oct 30 13:30:00.630282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.630229    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:00.636064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.636041    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"]
Oct 30 13:30:00.636180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.636074    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:00.636180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.636114    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"]
Oct 30 13:30:00.646429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.646394    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"]
Oct 30 13:30:00.652439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.652411    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:00.712501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.709746    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"]
Oct 30 13:30:00.712501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.711880    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rch\" (UniqueName: \"kubernetes.io/projected/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-kube-api-access-98rch\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6\" (UID: \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:00.715496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.714098    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972qg\" (UniqueName: \"kubernetes.io/projected/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-kube-api-access-972qg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6\" (UID: \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:00.737190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.737149    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrr6m\" (UniqueName: \"kubernetes.io/projected/86002868-e959-4e93-9a73-a6ca9695a40b-kube-api-access-mrr6m\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp\" (UID: \"86002868-e959-4e93-9a73-a6ca9695a40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:00.737390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.737224    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9\" (UID: \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:00.737390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.737328    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvz5d\" (UniqueName: \"kubernetes.io/projected/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-kube-api-access-dvz5d\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9\" (UID: \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:00.737390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.737374    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c140cb-a654-47db-b51e-a88e9455d374-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj\" (UID: \"77c140cb-a654-47db-b51e-a88e9455d374\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:00.737558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.737412    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18ca0c2-40ad-4793-982f-176d4efbf373-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4\" (UID: \"c18ca0c2-40ad-4793-982f-176d4efbf373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:00.737558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.737539    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz52t\" (UniqueName: \"kubernetes.io/projected/77c140cb-a654-47db-b51e-a88e9455d374-kube-api-access-bz52t\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj\" (UID: \"77c140cb-a654-47db-b51e-a88e9455d374\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:00.737663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.737613    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8mh\" (UniqueName: \"kubernetes.io/projected/6caff419-28cc-4098-a183-45af8e982db0-kube-api-access-4h8mh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx\" (UID: \"6caff419-28cc-4098-a183-45af8e982db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:00.737663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.737645    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86002868-e959-4e93-9a73-a6ca9695a40b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp\" (UID: \"86002868-e959-4e93-9a73-a6ca9695a40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:00.738126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.738093    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a20603-e044-4279-a9e8-7c2c7d030a44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7\" (UID: \"78a20603-e044-4279-a9e8-7c2c7d030a44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:00.738272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.738171    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6caff419-28cc-4098-a183-45af8e982db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx\" (UID: \"6caff419-28cc-4098-a183-45af8e982db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:00.738272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.738231    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjhm\" (UniqueName: \"kubernetes.io/projected/c18ca0c2-40ad-4793-982f-176d4efbf373-kube-api-access-cwjhm\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4\" (UID: \"c18ca0c2-40ad-4793-982f-176d4efbf373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:00.738387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.738276    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzkf\" (UniqueName: \"kubernetes.io/projected/78a20603-e044-4279-a9e8-7c2c7d030a44-kube-api-access-vrzkf\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7\" (UID: \"78a20603-e044-4279-a9e8-7c2c7d030a44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:00.738387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.738372    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf87f\" (UniqueName: \"kubernetes.io/projected/5af816ef-1abb-4490-a7e0-4d1901c27223-kube-api-access-xf87f\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw\" (UID: \"5af816ef-1abb-4490-a7e0-4d1901c27223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:00.739567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.738799    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a20603-e044-4279-a9e8-7c2c7d030a44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7\" (UID: \"78a20603-e044-4279-a9e8-7c2c7d030a44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:00.743709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.743323    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"]
Oct 30 13:30:00.744115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.744094    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:00.750597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.750575    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"]
Oct 30 13:30:00.750711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.750659    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:00.774421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.774383    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"]
Oct 30 13:30:00.788345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.787934    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrzkf\" (UniqueName: \"kubernetes.io/projected/78a20603-e044-4279-a9e8-7c2c7d030a44-kube-api-access-vrzkf\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7\" (UID: \"78a20603-e044-4279-a9e8-7c2c7d030a44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:00.790310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.790283    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"]
Oct 30 13:30:00.795639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.795618    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:00.797297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.797266    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"]
Oct 30 13:30:00.798158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.798128    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z" event={"ID":"986a714d-458c-47dc-b878-79c74ffa9979","Type":"ContainerStarted","Data":"d81895fbafa1532039916d1f88e5ed8ebfb6a6f03c9956057735e84567e7830d"}
Oct 30 13:30:00.804505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.804470    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"]
Oct 30 13:30:00.811312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.809826    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:00.817474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.817420    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"]
Oct 30 13:30:00.827499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.824900    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"]
Oct 30 13:30:00.836730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.831560    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:00.836730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.833957    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"]
Oct 30 13:30:00.837822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.837800    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"]
Oct 30 13:30:00.839899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.839878    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz52t\" (UniqueName: \"kubernetes.io/projected/77c140cb-a654-47db-b51e-a88e9455d374-kube-api-access-bz52t\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj\" (UID: \"77c140cb-a654-47db-b51e-a88e9455d374\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:00.840015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.839937    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8mh\" (UniqueName: \"kubernetes.io/projected/6caff419-28cc-4098-a183-45af8e982db0-kube-api-access-4h8mh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx\" (UID: \"6caff419-28cc-4098-a183-45af8e982db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:00.840015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.839969    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86002868-e959-4e93-9a73-a6ca9695a40b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp\" (UID: \"86002868-e959-4e93-9a73-a6ca9695a40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:00.840015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840008    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wm6\" (UniqueName: \"kubernetes.io/projected/dd6514e0-bbbd-4933-836f-fc32631577ad-kube-api-access-q5wm6\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w\" (UID: \"dd6514e0-bbbd-4933-836f-fc32631577ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:00.840194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840081    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6caff419-28cc-4098-a183-45af8e982db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx\" (UID: \"6caff419-28cc-4098-a183-45af8e982db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:00.840194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840118    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjhm\" (UniqueName: \"kubernetes.io/projected/c18ca0c2-40ad-4793-982f-176d4efbf373-kube-api-access-cwjhm\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4\" (UID: \"c18ca0c2-40ad-4793-982f-176d4efbf373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:00.840326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840199    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrr6m\" (UniqueName: \"kubernetes.io/projected/86002868-e959-4e93-9a73-a6ca9695a40b-kube-api-access-mrr6m\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp\" (UID: \"86002868-e959-4e93-9a73-a6ca9695a40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:00.840326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840269    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9\" (UID: \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:00.840644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840623    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvz5d\" (UniqueName: \"kubernetes.io/projected/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-kube-api-access-dvz5d\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9\" (UID: \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:00.840723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840674    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6514e0-bbbd-4933-836f-fc32631577ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w\" (UID: \"dd6514e0-bbbd-4933-836f-fc32631577ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:00.840723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840710    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c140cb-a654-47db-b51e-a88e9455d374-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj\" (UID: \"77c140cb-a654-47db-b51e-a88e9455d374\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:00.840843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.840773    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18ca0c2-40ad-4793-982f-176d4efbf373-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4\" (UID: \"c18ca0c2-40ad-4793-982f-176d4efbf373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:00.841139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.841114    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9\" (UID: \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:00.842242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.842194    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c140cb-a654-47db-b51e-a88e9455d374-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj\" (UID: \"77c140cb-a654-47db-b51e-a88e9455d374\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:00.842335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.842218    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6caff419-28cc-4098-a183-45af8e982db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx\" (UID: \"6caff419-28cc-4098-a183-45af8e982db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:00.842468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.842451    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"]
Oct 30 13:30:00.842801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.842776    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18ca0c2-40ad-4793-982f-176d4efbf373-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4\" (UID: \"c18ca0c2-40ad-4793-982f-176d4efbf373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:00.842988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.842970    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86002868-e959-4e93-9a73-a6ca9695a40b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp\" (UID: \"86002868-e959-4e93-9a73-a6ca9695a40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:00.846702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.846688    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"]
Oct 30 13:30:00.850818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.850797    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"]
Oct 30 13:30:00.892908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.892874    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"]
Oct 30 13:30:00.899590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.899563    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"]
Oct 30 13:30:00.905464 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.905438    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"]
Oct 30 13:30:00.905588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.905563    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:00.911808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.911666    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:00.916192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.916159    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"]
Oct 30 13:30:00.922008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.921940    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"]
Oct 30 13:30:00.931352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.931315    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:00.943065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.943035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6514e0-bbbd-4933-836f-fc32631577ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w\" (UID: \"dd6514e0-bbbd-4933-836f-fc32631577ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:00.943517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.943493    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbztb\" (UniqueName: \"kubernetes.io/projected/75bc6576-0613-4b84-a683-ed8afd915111-kube-api-access-nbztb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd\" (UID: \"75bc6576-0613-4b84-a683-ed8afd915111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:00.943779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.943755    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wm6\" (UniqueName: \"kubernetes.io/projected/dd6514e0-bbbd-4933-836f-fc32631577ad-kube-api-access-q5wm6\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w\" (UID: \"dd6514e0-bbbd-4933-836f-fc32631577ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:00.943926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.943892    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6514e0-bbbd-4933-836f-fc32631577ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w\" (UID: \"dd6514e0-bbbd-4933-836f-fc32631577ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:00.944114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.944038    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfktq\" (UniqueName: \"kubernetes.io/projected/187f5533-755c-4fda-b817-7e1d2b97defc-kube-api-access-kfktq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4\" (UID: \"187f5533-755c-4fda-b817-7e1d2b97defc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:00.944707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.944572    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187f5533-755c-4fda-b817-7e1d2b97defc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4\" (UID: \"187f5533-755c-4fda-b817-7e1d2b97defc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:00.944707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.944610    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c2fb-83d5-4af2-b030-5368808f2c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx\" (UID: \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:00.944707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.944648    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45be2541-5839-4419-afe3-3a747093066a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz\" (UID: \"45be2541-5839-4419-afe3-3a747093066a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:00.944707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.944679    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bclb\" (UniqueName: \"kubernetes.io/projected/45be2541-5839-4419-afe3-3a747093066a-kube-api-access-2bclb\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz\" (UID: \"45be2541-5839-4419-afe3-3a747093066a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:00.945032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.944784    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgrs\" (UniqueName: \"kubernetes.io/projected/6cb2c2fb-83d5-4af2-b030-5368808f2c82-kube-api-access-8pgrs\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx\" (UID: \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:00.945247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.945226    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"]
Oct 30 13:30:00.945533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.945446    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkz9j\" (UniqueName: \"kubernetes.io/projected/465dbcb3-1211-4b3e-840c-19d89a3e119e-kube-api-access-kkz9j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5\" (UID: \"465dbcb3-1211-4b3e-840c-19d89a3e119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:00.945533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.945487    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75bc6576-0613-4b84-a683-ed8afd915111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd\" (UID: \"75bc6576-0613-4b84-a683-ed8afd915111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:00.945533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.945564    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465dbcb3-1211-4b3e-840c-19d89a3e119e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5\" (UID: \"465dbcb3-1211-4b3e-840c-19d89a3e119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:00.946352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.945810    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:00.977441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.976290    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:00.977441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.976989    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz52t\" (UniqueName: \"kubernetes.io/projected/77c140cb-a654-47db-b51e-a88e9455d374-kube-api-access-bz52t\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj\" (UID: \"77c140cb-a654-47db-b51e-a88e9455d374\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:00.984021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.983955    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8mh\" (UniqueName: \"kubernetes.io/projected/6caff419-28cc-4098-a183-45af8e982db0-kube-api-access-4h8mh\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx\" (UID: \"6caff419-28cc-4098-a183-45af8e982db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:00.986307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.985244    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrr6m\" (UniqueName: \"kubernetes.io/projected/86002868-e959-4e93-9a73-a6ca9695a40b-kube-api-access-mrr6m\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp\" (UID: \"86002868-e959-4e93-9a73-a6ca9695a40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:00.987601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.987579    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvz5d\" (UniqueName: \"kubernetes.io/projected/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-kube-api-access-dvz5d\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9\" (UID: \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:00.990010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.989977    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjhm\" (UniqueName: \"kubernetes.io/projected/c18ca0c2-40ad-4793-982f-176d4efbf373-kube-api-access-cwjhm\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4\" (UID: \"c18ca0c2-40ad-4793-982f-176d4efbf373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:00.992884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:00.992857    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wm6\" (UniqueName: \"kubernetes.io/projected/dd6514e0-bbbd-4933-836f-fc32631577ad-kube-api-access-q5wm6\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w\" (UID: \"dd6514e0-bbbd-4933-836f-fc32631577ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:01.014493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.014256    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:01.033250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.031769    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049020    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465dbcb3-1211-4b3e-840c-19d89a3e119e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5\" (UID: \"465dbcb3-1211-4b3e-840c-19d89a3e119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049134    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4\" (UID: \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049307    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbztb\" (UniqueName: \"kubernetes.io/projected/75bc6576-0613-4b84-a683-ed8afd915111-kube-api-access-nbztb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd\" (UID: \"75bc6576-0613-4b84-a683-ed8afd915111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049378    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfktq\" (UniqueName: \"kubernetes.io/projected/187f5533-755c-4fda-b817-7e1d2b97defc-kube-api-access-kfktq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4\" (UID: \"187f5533-755c-4fda-b817-7e1d2b97defc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049439    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187f5533-755c-4fda-b817-7e1d2b97defc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4\" (UID: \"187f5533-755c-4fda-b817-7e1d2b97defc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049468    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c2fb-83d5-4af2-b030-5368808f2c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx\" (UID: \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049499    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45be2541-5839-4419-afe3-3a747093066a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz\" (UID: \"45be2541-5839-4419-afe3-3a747093066a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049531    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fpw\" (UniqueName: \"kubernetes.io/projected/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-kube-api-access-p9fpw\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4\" (UID: \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049557    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bclb\" (UniqueName: \"kubernetes.io/projected/45be2541-5839-4419-afe3-3a747093066a-kube-api-access-2bclb\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz\" (UID: \"45be2541-5839-4419-afe3-3a747093066a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049579    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgrs\" (UniqueName: \"kubernetes.io/projected/6cb2c2fb-83d5-4af2-b030-5368808f2c82-kube-api-access-8pgrs\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx\" (UID: \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049617    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzgd\" (UniqueName: \"kubernetes.io/projected/d14cbc6e-1967-47a3-be75-b2251ecc25ce-kube-api-access-nmzgd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5\" (UID: \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049644    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14cbc6e-1967-47a3-be75-b2251ecc25ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5\" (UID: \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049671    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkz9j\" (UniqueName: \"kubernetes.io/projected/465dbcb3-1211-4b3e-840c-19d89a3e119e-kube-api-access-kkz9j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5\" (UID: \"465dbcb3-1211-4b3e-840c-19d89a3e119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.049697    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75bc6576-0613-4b84-a683-ed8afd915111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd\" (UID: \"75bc6576-0613-4b84-a683-ed8afd915111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.050483    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75bc6576-0613-4b84-a683-ed8afd915111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd\" (UID: \"75bc6576-0613-4b84-a683-ed8afd915111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:01.083799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.051142    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465dbcb3-1211-4b3e-840c-19d89a3e119e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5\" (UID: \"465dbcb3-1211-4b3e-840c-19d89a3e119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:01.098503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.089454    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187f5533-755c-4fda-b817-7e1d2b97defc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4\" (UID: \"187f5533-755c-4fda-b817-7e1d2b97defc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:01.098503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.089505    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45be2541-5839-4419-afe3-3a747093066a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz\" (UID: \"45be2541-5839-4419-afe3-3a747093066a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:01.098503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.090174    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c2fb-83d5-4af2-b030-5368808f2c82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx\" (UID: \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:01.141388 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.140904    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfktq\" (UniqueName: \"kubernetes.io/projected/187f5533-755c-4fda-b817-7e1d2b97defc-kube-api-access-kfktq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4\" (UID: \"187f5533-755c-4fda-b817-7e1d2b97defc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:01.148815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.142423    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:01.148815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.145622    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkz9j\" (UniqueName: \"kubernetes.io/projected/465dbcb3-1211-4b3e-840c-19d89a3e119e-kube-api-access-kkz9j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5\" (UID: \"465dbcb3-1211-4b3e-840c-19d89a3e119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.150176    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgrs\" (UniqueName: \"kubernetes.io/projected/6cb2c2fb-83d5-4af2-b030-5368808f2c82-kube-api-access-8pgrs\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx\" (UID: \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.193169    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbztb\" (UniqueName: \"kubernetes.io/projected/75bc6576-0613-4b84-a683-ed8afd915111-kube-api-access-nbztb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd\" (UID: \"75bc6576-0613-4b84-a683-ed8afd915111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.199670    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.200098    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.202007    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fpw\" (UniqueName: \"kubernetes.io/projected/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-kube-api-access-p9fpw\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4\" (UID: \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.202074    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzgd\" (UniqueName: \"kubernetes.io/projected/d14cbc6e-1967-47a3-be75-b2251ecc25ce-kube-api-access-nmzgd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5\" (UID: \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.202110    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14cbc6e-1967-47a3-be75-b2251ecc25ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5\" (UID: \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.202255    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4\" (UID: \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.203497    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:01.205170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.203599    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4\" (UID: \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:01.206864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.206828    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14cbc6e-1967-47a3-be75-b2251ecc25ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5\" (UID: \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:01.217521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.217467    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:01.225491 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.223833    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:01.245041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.245017    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:01.246664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.246634    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bclb\" (UniqueName: \"kubernetes.io/projected/45be2541-5839-4419-afe3-3a747093066a-kube-api-access-2bclb\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz\" (UID: \"45be2541-5839-4419-afe3-3a747093066a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:01.270967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.251647    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:01.292036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.291737    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:01.314335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.311800    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:01.314335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.311805    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzgd\" (UniqueName: \"kubernetes.io/projected/d14cbc6e-1967-47a3-be75-b2251ecc25ce-kube-api-access-nmzgd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5\" (UID: \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:01.314788 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.314686    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fpw\" (UniqueName: \"kubernetes.io/projected/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-kube-api-access-p9fpw\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4\" (UID: \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:01.324836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.324510    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:01.339468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.339403    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:01.483794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.483339    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 13:30:01.483794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.483408    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 13:30:01.629392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.626926    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"]
Oct 30 13:30:01.644449 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:30:01.644404    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a20603_e044_4279_a9e8_7c2c7d030a44.slice/crio-1f6fd120a2eeb2ba3336ec0431ede20847afee5128bfc6538492c7cf6db704c0 WatchSource:0}: Error finding container 1f6fd120a2eeb2ba3336ec0431ede20847afee5128bfc6538492c7cf6db704c0: Status 404 returned error can't find the container with id 1f6fd120a2eeb2ba3336ec0431ede20847afee5128bfc6538492c7cf6db704c0
Oct 30 13:30:01.870055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.869643    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z" event={"ID":"986a714d-458c-47dc-b878-79c74ffa9979","Type":"ContainerStarted","Data":"a59478b7e2aab9213b573c04a6bb7a4f90826cf2223f5b46eb04ddfe37918aff"}
Oct 30 13:30:01.919229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.911464    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" event={"ID":"78a20603-e044-4279-a9e8-7c2c7d030a44","Type":"ContainerStarted","Data":"1f6fd120a2eeb2ba3336ec0431ede20847afee5128bfc6538492c7cf6db704c0"}
Oct 30 13:30:01.938198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:01.938118    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z" podStartSLOduration=2.938080302 podStartE2EDuration="2.938080302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:01.93173332 +0000 UTC m=+3466.407452931" watchObservedRunningTime="2026-10-30 13:30:01.938080302 +0000 UTC m=+3466.413799912"
Oct 30 13:30:02.122257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.122223    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"]
Oct 30 13:30:02.138705 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:30:02.124350    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18ca0c2_40ad_4793_982f_176d4efbf373.slice/crio-0cf8fc18148140e8d0777df9124e81dd15b6ea3e3fbe9cba0a3ec7dfe1312f34 WatchSource:0}: Error finding container 0cf8fc18148140e8d0777df9124e81dd15b6ea3e3fbe9cba0a3ec7dfe1312f34: Status 404 returned error can't find the container with id 0cf8fc18148140e8d0777df9124e81dd15b6ea3e3fbe9cba0a3ec7dfe1312f34
Oct 30 13:30:02.217519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.216518    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"]
Oct 30 13:30:02.233693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.231725    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"]
Oct 30 13:30:02.236944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.236879    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"]
Oct 30 13:30:02.244918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.244887    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"]
Oct 30 13:30:02.414322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.414287    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"]
Oct 30 13:30:02.539273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.525105    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 13:30:02.539273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.525160    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 13:30:02.621334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.614377    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"]
Oct 30 13:30:02.621334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.617282    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"]
Oct 30 13:30:02.632818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.632673    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"]
Oct 30 13:30:02.642998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.640747    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"]
Oct 30 13:30:02.734997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.734971    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"]
Oct 30 13:30:02.750608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:02.750580    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"]
Oct 30 13:30:03.006401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.006323    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"]
Oct 30 13:30:03.043879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.043832    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"]
Oct 30 13:30:03.051121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.051000    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" event={"ID":"75bc6576-0613-4b84-a683-ed8afd915111","Type":"ContainerStarted","Data":"3dde33671b9bbf8311fad65bd5c4ce8b74187dea0a45bc4c6aa5a5c35abfe44e"}
Oct 30 13:30:03.106397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.106364    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" event={"ID":"6cb2c2fb-83d5-4af2-b030-5368808f2c82","Type":"ContainerStarted","Data":"07af08c7adee25383a4382ccacbbf8b43ca0f96116d1f1e2b780e31111866727"}
Oct 30 13:30:03.106909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.106888    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"]
Oct 30 13:30:03.109138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.109094    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" event={"ID":"c18ca0c2-40ad-4793-982f-176d4efbf373","Type":"ContainerStarted","Data":"1dac33d5825aef8dfd929cc75ffcb391548fd87e23fe597631e7774429db4f04"}
Oct 30 13:30:03.112471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.112451    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"]
Oct 30 13:30:03.112632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.112617    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" event={"ID":"c18ca0c2-40ad-4793-982f-176d4efbf373","Type":"ContainerStarted","Data":"0cf8fc18148140e8d0777df9124e81dd15b6ea3e3fbe9cba0a3ec7dfe1312f34"}
Oct 30 13:30:03.119134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.119111    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" event={"ID":"b0ec16c9-b636-4241-bca4-6bcaf05e9f11","Type":"ContainerStarted","Data":"c2c3eb2e9f3a9f99d710ab65d460c963867f82bb12392e337fdfbdced8d82a82"}
Oct 30 13:30:03.119390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.119331    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"]
Oct 30 13:30:03.121426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.121112    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" event={"ID":"77c140cb-a654-47db-b51e-a88e9455d374","Type":"ContainerStarted","Data":"9de740bbe7226c124a7c31d1413d0d92a3d5b9d3e35ad3a9b9c4c005c083bf98"}
Oct 30 13:30:03.122563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.122544    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" event={"ID":"dd6514e0-bbbd-4933-836f-fc32631577ad","Type":"ContainerStarted","Data":"2238808ec98c86e80d31a2ba9e39bf17150ac7ddf927031d900f1fbfb54f0541"}
Oct 30 13:30:03.124381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.124357    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" event={"ID":"6caff419-28cc-4098-a183-45af8e982db0","Type":"ContainerStarted","Data":"262cc2a9236e769f74b6523d245ad4e970ae0755d46a6c8fcfa896ef9de860a6"}
Oct 30 13:30:03.126093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.126075    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" event={"ID":"5af816ef-1abb-4490-a7e0-4d1901c27223","Type":"ContainerStarted","Data":"931d76de34c827c8c77ad4f0e4670eb0569ede2cb11e50a9e345a1317c55258a"}
Oct 30 13:30:03.129134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.129057    4993 generic.go:356] "Generic (PLEG): container finished" podID="986a714d-458c-47dc-b878-79c74ffa9979" containerID="a59478b7e2aab9213b573c04a6bb7a4f90826cf2223f5b46eb04ddfe37918aff" exitCode=0
Oct 30 13:30:03.129256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.129219    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z" event={"ID":"986a714d-458c-47dc-b878-79c74ffa9979","Type":"ContainerDied","Data":"a59478b7e2aab9213b573c04a6bb7a4f90826cf2223f5b46eb04ddfe37918aff"}
Oct 30 13:30:03.131580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.131302    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" event={"ID":"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3","Type":"ContainerStarted","Data":"b3a58dc0be87dc744651e46dc5c2ba8ef3a7ef8f47a64421b06173237cb3a135"}
Oct 30 13:30:03.131736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.131715    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" event={"ID":"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3","Type":"ContainerStarted","Data":"8e33c895021e7619a9781f46d36ae5c1ab25bbad69508c323599bc4de803ada4"}
Oct 30 13:30:03.133179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.133155    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" event={"ID":"78a20603-e044-4279-a9e8-7c2c7d030a44","Type":"ContainerStarted","Data":"62f94c6eb1acab69aa9c1bb64118c44933db8a28b12686dd28430b4bdd770f43"}
Oct 30 13:30:03.135006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.134978    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" event={"ID":"ce27ddf5-ef65-427e-8793-bfbc3015ddbe","Type":"ContainerStarted","Data":"7756cdd946cdde69a3f0d599192ce9acc33549b39a3141ce94fdb3c21be96223"}
Oct 30 13:30:03.136536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.136514    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" event={"ID":"86002868-e959-4e93-9a73-a6ca9695a40b","Type":"ContainerStarted","Data":"5361c8d16955485385674e628ec5b1fac91627ffb335862ea352d06f8933f076"}
Oct 30 13:30:03.169407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.165494    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" podStartSLOduration=3.165470595 podStartE2EDuration="3.165470595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.154844168 +0000 UTC m=+3467.630563778" watchObservedRunningTime="2026-10-30 13:30:03.165470595 +0000 UTC m=+3467.641190206"
Oct 30 13:30:03.495698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.492225    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 13:30:03.495698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.492285    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 13:30:03.541959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.541894    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" podStartSLOduration=3.541871885 podStartE2EDuration="3.541871885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.230075857 +0000 UTC m=+3467.705795469" watchObservedRunningTime="2026-10-30 13:30:03.541871885 +0000 UTC m=+3468.017591476"
Oct 30 13:30:03.623461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.623408    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:30:03.625807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.625765    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" podStartSLOduration=3.6257479139999997 podStartE2EDuration="3.625747914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.618351091 +0000 UTC m=+3468.094070702" watchObservedRunningTime="2026-10-30 13:30:03.625747914 +0000 UTC m=+3468.101467525"
Oct 30 13:30:03.708275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.708181    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" podStartSLOduration=3.708162622 podStartE2EDuration="3.708162622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.702474287 +0000 UTC m=+3468.178193898" watchObservedRunningTime="2026-10-30 13:30:03.708162622 +0000 UTC m=+3468.183882233"
Oct 30 13:30:03.789237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:03.785885    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" podStartSLOduration=3.785865259 podStartE2EDuration="3.785865259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.765804901 +0000 UTC m=+3468.241524511" watchObservedRunningTime="2026-10-30 13:30:03.785865259 +0000 UTC m=+3468.261584873"
Oct 30 13:30:04.148291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.147340    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" event={"ID":"75bc6576-0613-4b84-a683-ed8afd915111","Type":"ContainerStarted","Data":"48e40ade278497374732f140fcf0e5aeb0bf119fd8de7b50c992c724d382dc27"}
Oct 30 13:30:04.150762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.150729    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" event={"ID":"6cb2c2fb-83d5-4af2-b030-5368808f2c82","Type":"ContainerStarted","Data":"62cda6248d0e16ce5dabd01fbe97cd1ddba44777e9945db361615372c654f096"}
Oct 30 13:30:04.155351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.152320    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" event={"ID":"187f5533-755c-4fda-b817-7e1d2b97defc","Type":"ContainerStarted","Data":"86c8f5b3c56bc6cce1801635eecb0dd200b296d3fb885cfc5ae845bfa4c5a3f3"}
Oct 30 13:30:04.159724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.159691    4993 generic.go:356] "Generic (PLEG): container finished" podID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerID="1dac33d5825aef8dfd929cc75ffcb391548fd87e23fe597631e7774429db4f04" exitCode=60
Oct 30 13:30:04.160295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.160270    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" event={"ID":"c18ca0c2-40ad-4793-982f-176d4efbf373","Type":"ContainerDied","Data":"1dac33d5825aef8dfd929cc75ffcb391548fd87e23fe597631e7774429db4f04"}
Oct 30 13:30:04.161086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.161064    4993 scope.go:122] "RemoveContainer" containerID="1dac33d5825aef8dfd929cc75ffcb391548fd87e23fe597631e7774429db4f04"
Oct 30 13:30:04.183327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.183293    4993 generic.go:356] "Generic (PLEG): container finished" podID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerID="3931c6054b308859a5146ac46c6a46a629d9212421cdba45c07c6ee33e1ddae0" exitCode=60
Oct 30 13:30:04.183624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.183597    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" event={"ID":"b0ec16c9-b636-4241-bca4-6bcaf05e9f11","Type":"ContainerDied","Data":"3931c6054b308859a5146ac46c6a46a629d9212421cdba45c07c6ee33e1ddae0"}
Oct 30 13:30:04.191349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.184236    4993 scope.go:122] "RemoveContainer" containerID="3931c6054b308859a5146ac46c6a46a629d9212421cdba45c07c6ee33e1ddae0"
Oct 30 13:30:04.194239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.186709    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" event={"ID":"77c140cb-a654-47db-b51e-a88e9455d374","Type":"ContainerStarted","Data":"1c7c3e596109991b69d6ec4b487879d400e27b5e4944cb7528c6a13514cf20ef"}
Oct 30 13:30:04.215693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.215660    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" event={"ID":"465dbcb3-1211-4b3e-840c-19d89a3e119e","Type":"ContainerStarted","Data":"ae9413812d1b3c15a7a70466973bbb22af79d071c856137526ff0abba6aee947"}
Oct 30 13:30:04.217099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.217003    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" event={"ID":"45be2541-5839-4419-afe3-3a747093066a","Type":"ContainerStarted","Data":"817188b66efde3d04eeb5e52fa870b279a86622c4432c29580104ba063dd3a36"}
Oct 30 13:30:04.220082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.220060    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" event={"ID":"dd6514e0-bbbd-4933-836f-fc32631577ad","Type":"ContainerStarted","Data":"cc53cdb0cc2320bdf5bb0980e6ad8c4c803e6d935436197c51ded40ac1fae183"}
Oct 30 13:30:04.220579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.220384    4993 scope.go:122] "RemoveContainer" containerID="cc53cdb0cc2320bdf5bb0980e6ad8c4c803e6d935436197c51ded40ac1fae183"
Oct 30 13:30:04.222616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.222595    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" event={"ID":"6caff419-28cc-4098-a183-45af8e982db0","Type":"ContainerStarted","Data":"f32e2eaaa623b3d7fedcbd2a04038ef6e519fa3584d9ab15bf8f2b9b6e76a76c"}
Oct 30 13:30:04.225127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.225104    4993 generic.go:356] "Generic (PLEG): container finished" podID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerID="7650a6d63f523ff12b9d0ca2bc1a802564202b76a6c71f1948c79e02b3f8935c" exitCode=60
Oct 30 13:30:04.225225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.225179    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" event={"ID":"5af816ef-1abb-4490-a7e0-4d1901c27223","Type":"ContainerDied","Data":"7650a6d63f523ff12b9d0ca2bc1a802564202b76a6c71f1948c79e02b3f8935c"}
Oct 30 13:30:04.225506 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.225490    4993 scope.go:122] "RemoveContainer" containerID="7650a6d63f523ff12b9d0ca2bc1a802564202b76a6c71f1948c79e02b3f8935c"
Oct 30 13:30:04.226878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.226836    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" event={"ID":"bdcaa358-f46d-4b45-97bb-3e617acd4ed8","Type":"ContainerStarted","Data":"152ec0bd597addd059e273f805bdffbbfd483939bcd124a1f253d6d9c4167f78"}
Oct 30 13:30:04.229351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.229326    4993 generic.go:356] "Generic (PLEG): container finished" podID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerID="b3a58dc0be87dc744651e46dc5c2ba8ef3a7ef8f47a64421b06173237cb3a135" exitCode=60
Oct 30 13:30:04.229440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.229419    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" event={"ID":"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3","Type":"ContainerDied","Data":"b3a58dc0be87dc744651e46dc5c2ba8ef3a7ef8f47a64421b06173237cb3a135"}
Oct 30 13:30:04.229677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.229664    4993 scope.go:122] "RemoveContainer" containerID="b3a58dc0be87dc744651e46dc5c2ba8ef3a7ef8f47a64421b06173237cb3a135"
Oct 30 13:30:04.236549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.236527    4993 generic.go:356] "Generic (PLEG): container finished" podID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerID="62f94c6eb1acab69aa9c1bb64118c44933db8a28b12686dd28430b4bdd770f43" exitCode=60
Oct 30 13:30:04.236642 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.236558    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" event={"ID":"78a20603-e044-4279-a9e8-7c2c7d030a44","Type":"ContainerDied","Data":"62f94c6eb1acab69aa9c1bb64118c44933db8a28b12686dd28430b4bdd770f43"}
Oct 30 13:30:04.237330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.237299    4993 scope.go:122] "RemoveContainer" containerID="62f94c6eb1acab69aa9c1bb64118c44933db8a28b12686dd28430b4bdd770f43"
Oct 30 13:30:04.238937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.238914    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" event={"ID":"ce27ddf5-ef65-427e-8793-bfbc3015ddbe","Type":"ContainerStarted","Data":"5eaefac224df5dd13919bf6cad62b1f1a417ecfa61eb29e653b90e9d4c8d2287"}
Oct 30 13:30:04.245992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.245960    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" event={"ID":"d14cbc6e-1967-47a3-be75-b2251ecc25ce","Type":"ContainerStarted","Data":"22cb5620364fc6044163247d95c9b9ea8121684372823a0c11f416692e428dfc"}
Oct 30 13:30:04.275562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.275503    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" podStartSLOduration=4.275481624 podStartE2EDuration="4.275481624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.207917721 +0000 UTC m=+3468.683637335" watchObservedRunningTime="2026-10-30 13:30:04.275481624 +0000 UTC m=+3468.751201236"
Oct 30 13:30:04.337389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.336425    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" podStartSLOduration=4.336407466 podStartE2EDuration="4.336407466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.334108806 +0000 UTC m=+3468.809828415" watchObservedRunningTime="2026-10-30 13:30:04.336407466 +0000 UTC m=+3468.812127077"
Oct 30 13:30:04.452091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.452040    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" podStartSLOduration=4.452023435 podStartE2EDuration="4.452023435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.449066699 +0000 UTC m=+3468.924786318" watchObservedRunningTime="2026-10-30 13:30:04.452023435 +0000 UTC m=+3468.927743045"
Oct 30 13:30:04.499330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.498891    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 13:30:04.499330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.498949    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 13:30:04.696302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.670087    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" podStartSLOduration=4.670067691 podStartE2EDuration="4.670067691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.538952441 +0000 UTC m=+3469.014672074" watchObservedRunningTime="2026-10-30 13:30:04.670067691 +0000 UTC m=+3469.145787304"
Oct 30 13:30:04.823998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.818312    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" podStartSLOduration=4.818293222 podStartE2EDuration="4.818293222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.817763523 +0000 UTC m=+3469.293483134" watchObservedRunningTime="2026-10-30 13:30:04.818293222 +0000 UTC m=+3469.294012833"
Oct 30 13:30:04.887230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:04.883406    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:30:05.100151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.100095    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986a714d-458c-47dc-b878-79c74ffa9979-kube-api-access-v46q4\" (UniqueName: \"kubernetes.io/projected/986a714d-458c-47dc-b878-79c74ffa9979-kube-api-access-v46q4\") pod \"986a714d-458c-47dc-b878-79c74ffa9979\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") "
Oct 30 13:30:05.100890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.100259    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-xks-ca\") pod \"986a714d-458c-47dc-b878-79c74ffa9979\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") "
Oct 30 13:30:05.100890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.100409    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-openshift-ca\") pod \"986a714d-458c-47dc-b878-79c74ffa9979\" (UID: \"986a714d-458c-47dc-b878-79c74ffa9979\") "
Oct 30 13:30:05.101550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.101505    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-openshift-ca" pod "986a714d-458c-47dc-b878-79c74ffa9979" (UID: "986a714d-458c-47dc-b878-79c74ffa9979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:05.109584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.109536    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986a714d-458c-47dc-b878-79c74ffa9979-kube-api-access-v46q4" pod "986a714d-458c-47dc-b878-79c74ffa9979" (UID: "986a714d-458c-47dc-b878-79c74ffa9979"). InnerVolumeSpecName "kube-api-access-v46q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:05.112318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.111774    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-xks-ca" pod "986a714d-458c-47dc-b878-79c74ffa9979" (UID: "986a714d-458c-47dc-b878-79c74ffa9979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:30:05.203088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.203008    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46q4\" (UniqueName: \"kubernetes.io/projected/986a714d-458c-47dc-b878-79c74ffa9979-kube-api-access-v46q4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:05.203088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.203042    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:05.203088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.203055    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986a714d-458c-47dc-b878-79c74ffa9979-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:05.260939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.258984    4993 generic.go:356] "Generic (PLEG): container finished" podID="75bc6576-0613-4b84-a683-ed8afd915111" containerID="48e40ade278497374732f140fcf0e5aeb0bf119fd8de7b50c992c724d382dc27" exitCode=60
Oct 30 13:30:05.260939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.259072    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" event={"ID":"75bc6576-0613-4b84-a683-ed8afd915111","Type":"ContainerDied","Data":"48e40ade278497374732f140fcf0e5aeb0bf119fd8de7b50c992c724d382dc27"}
Oct 30 13:30:05.260939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.259586    4993 scope.go:122] "RemoveContainer" containerID="48e40ade278497374732f140fcf0e5aeb0bf119fd8de7b50c992c724d382dc27"
Oct 30 13:30:05.264795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.264762    4993 generic.go:356] "Generic (PLEG): container finished" podID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerID="62cda6248d0e16ce5dabd01fbe97cd1ddba44777e9945db361615372c654f096" exitCode=60
Oct 30 13:30:05.264943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.264925    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" event={"ID":"6cb2c2fb-83d5-4af2-b030-5368808f2c82","Type":"ContainerDied","Data":"62cda6248d0e16ce5dabd01fbe97cd1ddba44777e9945db361615372c654f096"}
Oct 30 13:30:05.265271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.265253    4993 scope.go:122] "RemoveContainer" containerID="62cda6248d0e16ce5dabd01fbe97cd1ddba44777e9945db361615372c654f096"
Oct 30 13:30:05.272452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.272331    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" event={"ID":"187f5533-755c-4fda-b817-7e1d2b97defc","Type":"ContainerStarted","Data":"2d8f967aea32b397b20056df26e464f4f94b29f20ae49407707e266c6d2c0923"}
Oct 30 13:30:05.275433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.274732    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" event={"ID":"c18ca0c2-40ad-4793-982f-176d4efbf373","Type":"ContainerStarted","Data":"d976d98ea937191126d788b4401c937179fbd322e92337e5bf9aef35ddcfe2a9"}
Oct 30 13:30:05.277637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.277479    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" event={"ID":"b0ec16c9-b636-4241-bca4-6bcaf05e9f11","Type":"ContainerStarted","Data":"8ca83df01c00eac4739cddc6fca5aab610fcfaf5e526c99782fc599b4d4fa3f7"}
Oct 30 13:30:05.279775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.279749    4993 generic.go:356] "Generic (PLEG): container finished" podID="77c140cb-a654-47db-b51e-a88e9455d374" containerID="1c7c3e596109991b69d6ec4b487879d400e27b5e4944cb7528c6a13514cf20ef" exitCode=60
Oct 30 13:30:05.279879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.279833    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" event={"ID":"77c140cb-a654-47db-b51e-a88e9455d374","Type":"ContainerDied","Data":"1c7c3e596109991b69d6ec4b487879d400e27b5e4944cb7528c6a13514cf20ef"}
Oct 30 13:30:05.280577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.280148    4993 scope.go:122] "RemoveContainer" containerID="1c7c3e596109991b69d6ec4b487879d400e27b5e4944cb7528c6a13514cf20ef"
Oct 30 13:30:05.289182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.288795    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" event={"ID":"465dbcb3-1211-4b3e-840c-19d89a3e119e","Type":"ContainerStarted","Data":"231e5e7dc9207b841003a9b56a06c126fee10b8f72496ba59d6664cd076c09a3"}
Oct 30 13:30:05.291698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.290966    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" event={"ID":"45be2541-5839-4419-afe3-3a747093066a","Type":"ContainerStarted","Data":"eb3c51cbcb2af3dec144e1df54e036a0a957ff006b308a1e9b220329ff1a3a2b"}
Oct 30 13:30:05.300679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.300647    4993 generic.go:356] "Generic (PLEG): container finished" podID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerID="cc53cdb0cc2320bdf5bb0980e6ad8c4c803e6d935436197c51ded40ac1fae183" exitCode=60
Oct 30 13:30:05.300801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.300769    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" event={"ID":"dd6514e0-bbbd-4933-836f-fc32631577ad","Type":"ContainerDied","Data":"cc53cdb0cc2320bdf5bb0980e6ad8c4c803e6d935436197c51ded40ac1fae183"}
Oct 30 13:30:05.300860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.300817    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" event={"ID":"dd6514e0-bbbd-4933-836f-fc32631577ad","Type":"ContainerStarted","Data":"a879d6c08a11f46d26a8acce11d9b140f611475d7faa6811157600b31fd51bee"}
Oct 30 13:30:05.305936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.305910    4993 generic.go:356] "Generic (PLEG): container finished" podID="6caff419-28cc-4098-a183-45af8e982db0" containerID="f32e2eaaa623b3d7fedcbd2a04038ef6e519fa3584d9ab15bf8f2b9b6e76a76c" exitCode=60
Oct 30 13:30:05.306079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.305989    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" event={"ID":"6caff419-28cc-4098-a183-45af8e982db0","Type":"ContainerDied","Data":"f32e2eaaa623b3d7fedcbd2a04038ef6e519fa3584d9ab15bf8f2b9b6e76a76c"}
Oct 30 13:30:05.306361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.306343    4993 scope.go:122] "RemoveContainer" containerID="f32e2eaaa623b3d7fedcbd2a04038ef6e519fa3584d9ab15bf8f2b9b6e76a76c"
Oct 30 13:30:05.313707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.313680    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" event={"ID":"5af816ef-1abb-4490-a7e0-4d1901c27223","Type":"ContainerStarted","Data":"8af234c59554a1c9d00d3452d0418dbf245da22fba5a021b060ae919cc22e6cc"}
Oct 30 13:30:05.328440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.328411    4993 generic.go:356] "Generic (PLEG): container finished" podID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerID="950959c1dd3057f47478af38f2758d56e041b99d22650e66c04aa88fa952929a" exitCode=60
Oct 30 13:30:05.328682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.328662    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" event={"ID":"bdcaa358-f46d-4b45-97bb-3e617acd4ed8","Type":"ContainerDied","Data":"950959c1dd3057f47478af38f2758d56e041b99d22650e66c04aa88fa952929a"}
Oct 30 13:30:05.329127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.329108    4993 scope.go:122] "RemoveContainer" containerID="950959c1dd3057f47478af38f2758d56e041b99d22650e66c04aa88fa952929a"
Oct 30 13:30:05.330659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.330635    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z" event={"ID":"986a714d-458c-47dc-b878-79c74ffa9979","Type":"ContainerDied","Data":"d81895fbafa1532039916d1f88e5ed8ebfb6a6f03c9956057735e84567e7830d"}
Oct 30 13:30:05.330762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.330680    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81895fbafa1532039916d1f88e5ed8ebfb6a6f03c9956057735e84567e7830d"
Oct 30 13:30:05.330762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.330737    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"
Oct 30 13:30:05.340367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.340348    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" event={"ID":"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3","Type":"ContainerStarted","Data":"d4abdc14531fb0b28898dac5902e55ff844abfb1ccf73895817b4d8a50fe1cc4"}
Oct 30 13:30:05.345127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.345107    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" event={"ID":"78a20603-e044-4279-a9e8-7c2c7d030a44","Type":"ContainerStarted","Data":"b2bba50eca92c9ef9bc8e332bc6e3d1f8f4d5a61a199ee803c073b84011071e8"}
Oct 30 13:30:05.347165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.347145    4993 generic.go:356] "Generic (PLEG): container finished" podID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerID="5eaefac224df5dd13919bf6cad62b1f1a417ecfa61eb29e653b90e9d4c8d2287" exitCode=60
Oct 30 13:30:05.347389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.347352    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" event={"ID":"ce27ddf5-ef65-427e-8793-bfbc3015ddbe","Type":"ContainerDied","Data":"5eaefac224df5dd13919bf6cad62b1f1a417ecfa61eb29e653b90e9d4c8d2287"}
Oct 30 13:30:05.348323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.348306    4993 scope.go:122] "RemoveContainer" containerID="5eaefac224df5dd13919bf6cad62b1f1a417ecfa61eb29e653b90e9d4c8d2287"
Oct 30 13:30:05.349231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.349186    4993 generic.go:356] "Generic (PLEG): container finished" podID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerID="55e3800914288f97009f2cf172d1fe8fa6534d003373d0671b86dc2cfdda2c95" exitCode=60
Oct 30 13:30:05.349442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.349267    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" event={"ID":"d14cbc6e-1967-47a3-be75-b2251ecc25ce","Type":"ContainerDied","Data":"55e3800914288f97009f2cf172d1fe8fa6534d003373d0671b86dc2cfdda2c95"}
Oct 30 13:30:05.349643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.349501    4993 scope.go:122] "RemoveContainer" containerID="55e3800914288f97009f2cf172d1fe8fa6534d003373d0671b86dc2cfdda2c95"
Oct 30 13:30:05.351248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.351172    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" event={"ID":"86002868-e959-4e93-9a73-a6ca9695a40b","Type":"ContainerStarted","Data":"e28aa5ef99394a14358e5cb43b34861dc374feaf5224d7d705c725e5a04d02c9"}
Oct 30 13:30:05.414463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.414410    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" podStartSLOduration=5.414396368 podStartE2EDuration="5.414396368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.411089618 +0000 UTC m=+3469.886809230" watchObservedRunningTime="2026-10-30 13:30:05.414396368 +0000 UTC m=+3469.890115978"
Oct 30 13:30:05.459020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.458974    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 13:30:05.459152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.459046    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 13:30:05.523398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.523350    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" podStartSLOduration=5.5233352 podStartE2EDuration="5.5233352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.52012943 +0000 UTC m=+3469.995849041" watchObservedRunningTime="2026-10-30 13:30:05.5233352 +0000 UTC m=+3469.999054809"
Oct 30 13:30:05.566875 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.566811    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" podStartSLOduration=5.566792393 podStartE2EDuration="5.566792393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.561331363 +0000 UTC m=+3470.037050973" watchObservedRunningTime="2026-10-30 13:30:05.566792393 +0000 UTC m=+3470.042512003"
Oct 30 13:30:05.802513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:05.802455    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" podStartSLOduration=5.802432501 podStartE2EDuration="5.802432501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.752101857 +0000 UTC m=+3470.227821468" watchObservedRunningTime="2026-10-30 13:30:05.802432501 +0000 UTC m=+3470.278152111"
Oct 30 13:30:06.001162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.001104    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" podStartSLOduration=6.00108535 podStartE2EDuration="6.00108535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.958742611 +0000 UTC m=+3470.434462222" watchObservedRunningTime="2026-10-30 13:30:06.00108535 +0000 UTC m=+3470.476804960"
Oct 30 13:30:06.441282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.430292    4993 generic.go:356] "Generic (PLEG): container finished" podID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerID="b2bba50eca92c9ef9bc8e332bc6e3d1f8f4d5a61a199ee803c073b84011071e8" exitCode=60
Oct 30 13:30:06.441282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.430426    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" event={"ID":"78a20603-e044-4279-a9e8-7c2c7d030a44","Type":"ContainerDied","Data":"b2bba50eca92c9ef9bc8e332bc6e3d1f8f4d5a61a199ee803c073b84011071e8"}
Oct 30 13:30:06.441282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.430515    4993 scope.go:122] "RemoveContainer" containerID="62f94c6eb1acab69aa9c1bb64118c44933db8a28b12686dd28430b4bdd770f43"
Oct 30 13:30:06.441282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.431027    4993 scope.go:122] "RemoveContainer" containerID="b2bba50eca92c9ef9bc8e332bc6e3d1f8f4d5a61a199ee803c073b84011071e8"
Oct 30 13:30:06.452247 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:06.446062    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7_odh-ai-gateway-infra(78a20603-e044-4279-a9e8-7c2c7d030a44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44"
Oct 30 13:30:06.461887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.461004    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 13:30:06.461887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.461063    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 13:30:06.478007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.465873    4993 generic.go:356] "Generic (PLEG): container finished" podID="86002868-e959-4e93-9a73-a6ca9695a40b" containerID="e28aa5ef99394a14358e5cb43b34861dc374feaf5224d7d705c725e5a04d02c9" exitCode=60
Oct 30 13:30:06.478007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.466056    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" event={"ID":"86002868-e959-4e93-9a73-a6ca9695a40b","Type":"ContainerDied","Data":"e28aa5ef99394a14358e5cb43b34861dc374feaf5224d7d705c725e5a04d02c9"}
Oct 30 13:30:06.478007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.466544    4993 scope.go:122] "RemoveContainer" containerID="e28aa5ef99394a14358e5cb43b34861dc374feaf5224d7d705c725e5a04d02c9"
Oct 30 13:30:06.729278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.723878    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" event={"ID":"75bc6576-0613-4b84-a683-ed8afd915111","Type":"ContainerStarted","Data":"4bc8d496751e391bd8c8206061ae98016f8565a61af94d38ef53ad67f88206b1"}
Oct 30 13:30:06.770957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.763953    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" event={"ID":"6cb2c2fb-83d5-4af2-b030-5368808f2c82","Type":"ContainerStarted","Data":"a903cce12038a206e424496c9acc620c2f056e5ee5475185e0984a45a630ee73"}
Oct 30 13:30:06.883514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.881330    4993 generic.go:356] "Generic (PLEG): container finished" podID="187f5533-755c-4fda-b817-7e1d2b97defc" containerID="2d8f967aea32b397b20056df26e464f4f94b29f20ae49407707e266c6d2c0923" exitCode=60
Oct 30 13:30:06.883514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.881474    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" event={"ID":"187f5533-755c-4fda-b817-7e1d2b97defc","Type":"ContainerDied","Data":"2d8f967aea32b397b20056df26e464f4f94b29f20ae49407707e266c6d2c0923"}
Oct 30 13:30:06.883514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.882058    4993 scope.go:122] "RemoveContainer" containerID="2d8f967aea32b397b20056df26e464f4f94b29f20ae49407707e266c6d2c0923"
Oct 30 13:30:06.978494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.976817    4993 generic.go:356] "Generic (PLEG): container finished" podID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerID="d976d98ea937191126d788b4401c937179fbd322e92337e5bf9aef35ddcfe2a9" exitCode=60
Oct 30 13:30:06.978494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.976918    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" event={"ID":"c18ca0c2-40ad-4793-982f-176d4efbf373","Type":"ContainerDied","Data":"d976d98ea937191126d788b4401c937179fbd322e92337e5bf9aef35ddcfe2a9"}
Oct 30 13:30:06.978494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.976974    4993 scope.go:122] "RemoveContainer" containerID="1dac33d5825aef8dfd929cc75ffcb391548fd87e23fe597631e7774429db4f04"
Oct 30 13:30:06.978494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.977446    4993 scope.go:122] "RemoveContainer" containerID="d976d98ea937191126d788b4401c937179fbd322e92337e5bf9aef35ddcfe2a9"
Oct 30 13:30:06.978494 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:06.977837    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4_odh-ai-gateway-infra(c18ca0c2-40ad-4793-982f-176d4efbf373)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373"
Oct 30 13:30:06.987233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.987009    4993 generic.go:356] "Generic (PLEG): container finished" podID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerID="8ca83df01c00eac4739cddc6fca5aab610fcfaf5e526c99782fc599b4d4fa3f7" exitCode=60
Oct 30 13:30:06.987233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.987111    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" event={"ID":"b0ec16c9-b636-4241-bca4-6bcaf05e9f11","Type":"ContainerDied","Data":"8ca83df01c00eac4739cddc6fca5aab610fcfaf5e526c99782fc599b4d4fa3f7"}
Oct 30 13:30:06.988465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.987955    4993 scope.go:122] "RemoveContainer" containerID="8ca83df01c00eac4739cddc6fca5aab610fcfaf5e526c99782fc599b4d4fa3f7"
Oct 30 13:30:06.988465 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:06.988398    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6_odh-ai-gateway-infra(b0ec16c9-b636-4241-bca4-6bcaf05e9f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11"
Oct 30 13:30:06.991793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.991763    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" event={"ID":"77c140cb-a654-47db-b51e-a88e9455d374","Type":"ContainerStarted","Data":"47993b406feb4667333cd8d96f3d77be466d29ff5fbf195525d016db0414255d"}
Oct 30 13:30:06.992716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.992272    4993 scope.go:122] "RemoveContainer" containerID="47993b406feb4667333cd8d96f3d77be466d29ff5fbf195525d016db0414255d"
Oct 30 13:30:06.992716 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:06.992666    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj_odh-ai-gateway-infra(77c140cb-a654-47db-b51e-a88e9455d374)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" podUID="77c140cb-a654-47db-b51e-a88e9455d374"
Oct 30 13:30:06.995379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.995243    4993 generic.go:356] "Generic (PLEG): container finished" podID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerID="231e5e7dc9207b841003a9b56a06c126fee10b8f72496ba59d6664cd076c09a3" exitCode=60
Oct 30 13:30:06.995379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.995322    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" event={"ID":"465dbcb3-1211-4b3e-840c-19d89a3e119e","Type":"ContainerDied","Data":"231e5e7dc9207b841003a9b56a06c126fee10b8f72496ba59d6664cd076c09a3"}
Oct 30 13:30:06.995768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.995747    4993 scope.go:122] "RemoveContainer" containerID="231e5e7dc9207b841003a9b56a06c126fee10b8f72496ba59d6664cd076c09a3"
Oct 30 13:30:06.999179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.999159    4993 generic.go:356] "Generic (PLEG): container finished" podID="45be2541-5839-4419-afe3-3a747093066a" containerID="eb3c51cbcb2af3dec144e1df54e036a0a957ff006b308a1e9b220329ff1a3a2b" exitCode=60
Oct 30 13:30:07.000293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:06.999320    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" event={"ID":"45be2541-5839-4419-afe3-3a747093066a","Type":"ContainerDied","Data":"eb3c51cbcb2af3dec144e1df54e036a0a957ff006b308a1e9b220329ff1a3a2b"}
Oct 30 13:30:07.010226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.007546    4993 scope.go:122] "RemoveContainer" containerID="eb3c51cbcb2af3dec144e1df54e036a0a957ff006b308a1e9b220329ff1a3a2b"
Oct 30 13:30:07.012646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.011593    4993 generic.go:356] "Generic (PLEG): container finished" podID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerID="a879d6c08a11f46d26a8acce11d9b140f611475d7faa6811157600b31fd51bee" exitCode=60
Oct 30 13:30:07.012646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.011690    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" event={"ID":"dd6514e0-bbbd-4933-836f-fc32631577ad","Type":"ContainerDied","Data":"a879d6c08a11f46d26a8acce11d9b140f611475d7faa6811157600b31fd51bee"}
Oct 30 13:30:07.012646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.012070    4993 scope.go:122] "RemoveContainer" containerID="a879d6c08a11f46d26a8acce11d9b140f611475d7faa6811157600b31fd51bee"
Oct 30 13:30:07.012646 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:07.012581    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w_odh-ai-gateway-infra(dd6514e0-bbbd-4933-836f-fc32631577ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad"
Oct 30 13:30:07.017252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.016190    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" event={"ID":"6caff419-28cc-4098-a183-45af8e982db0","Type":"ContainerStarted","Data":"489ad9e6d6ab10d35f2c16bac0061559101a45ab4bc26fff682a723d1a86f1bc"}
Oct 30 13:30:07.018050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.017544    4993 scope.go:122] "RemoveContainer" containerID="489ad9e6d6ab10d35f2c16bac0061559101a45ab4bc26fff682a723d1a86f1bc"
Oct 30 13:30:07.018050 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:07.017953    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx_odh-ai-gateway-infra(6caff419-28cc-4098-a183-45af8e982db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" podUID="6caff419-28cc-4098-a183-45af8e982db0"
Oct 30 13:30:07.027589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.027506    4993 generic.go:356] "Generic (PLEG): container finished" podID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerID="8af234c59554a1c9d00d3452d0418dbf245da22fba5a021b060ae919cc22e6cc" exitCode=60
Oct 30 13:30:07.027747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.027674    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" event={"ID":"5af816ef-1abb-4490-a7e0-4d1901c27223","Type":"ContainerDied","Data":"8af234c59554a1c9d00d3452d0418dbf245da22fba5a021b060ae919cc22e6cc"}
Oct 30 13:30:07.028006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.027983    4993 scope.go:122] "RemoveContainer" containerID="8af234c59554a1c9d00d3452d0418dbf245da22fba5a021b060ae919cc22e6cc"
Oct 30 13:30:07.031668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.031406    4993 scope.go:122] "RemoveContainer" containerID="3931c6054b308859a5146ac46c6a46a629d9212421cdba45c07c6ee33e1ddae0"
Oct 30 13:30:07.032188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.032108    4993 generic.go:356] "Generic (PLEG): container finished" podID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerID="d4abdc14531fb0b28898dac5902e55ff844abfb1ccf73895817b4d8a50fe1cc4" exitCode=60
Oct 30 13:30:07.032188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.032152    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" event={"ID":"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3","Type":"ContainerDied","Data":"d4abdc14531fb0b28898dac5902e55ff844abfb1ccf73895817b4d8a50fe1cc4"}
Oct 30 13:30:07.033125 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:07.032579    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw_odh-ai-gateway-infra(5af816ef-1abb-4490-a7e0-4d1901c27223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223"
Oct 30 13:30:07.033125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.032817    4993 scope.go:122] "RemoveContainer" containerID="d4abdc14531fb0b28898dac5902e55ff844abfb1ccf73895817b4d8a50fe1cc4"
Oct 30 13:30:07.033507 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:07.033455    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6_odh-ai-gateway-infra(4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3"
Oct 30 13:30:07.098134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.098097    4993 scope.go:122] "RemoveContainer" containerID="cc53cdb0cc2320bdf5bb0980e6ad8c4c803e6d935436197c51ded40ac1fae183"
Oct 30 13:30:07.116668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.116637    4993 scope.go:122] "RemoveContainer" containerID="7650a6d63f523ff12b9d0ca2bc1a802564202b76a6c71f1948c79e02b3f8935c"
Oct 30 13:30:07.133179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.133155    4993 scope.go:122] "RemoveContainer" containerID="b3a58dc0be87dc744651e46dc5c2ba8ef3a7ef8f47a64421b06173237cb3a135"
Oct 30 13:30:07.245672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.245548    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6" podUID="1532e5a1-3417-487b-95b4-947fb71b20a4" containerName="payload-processing" containerID="cri-o://4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d" gracePeriod=25
Oct 30 13:30:07.485102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.485061    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 30 13:30:07.485586 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.485291    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 30 13:30:07.613474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.613447    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:30:07.818052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.817969    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-provider-credentials\") pod \"1532e5a1-3417-487b-95b4-947fb71b20a4\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") "
Oct 30 13:30:07.818637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.818613    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-plugins-config-volume\") pod \"1532e5a1-3417-487b-95b4-947fb71b20a4\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") "
Oct 30 13:30:07.818842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.818825    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-routing-overlay-volume\") pod \"1532e5a1-3417-487b-95b4-947fb71b20a4\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") "
Oct 30 13:30:07.819150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.819130    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-kube-api-access-btm5k\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-kube-api-access-btm5k\") pod \"1532e5a1-3417-487b-95b4-947fb71b20a4\" (UID: \"1532e5a1-3417-487b-95b4-947fb71b20a4\") "
Oct 30 13:30:07.820700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.820427    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-plugins-config-volume" pod "1532e5a1-3417-487b-95b4-947fb71b20a4" (UID: "1532e5a1-3417-487b-95b4-947fb71b20a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:07.821244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.820941    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-routing-overlay-volume" pod "1532e5a1-3417-487b-95b4-947fb71b20a4" (UID: "1532e5a1-3417-487b-95b4-947fb71b20a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:07.825659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.825638    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-kube-api-access-btm5k" pod "1532e5a1-3417-487b-95b4-947fb71b20a4" (UID: "1532e5a1-3417-487b-95b4-947fb71b20a4"). InnerVolumeSpecName "kube-api-access-btm5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:07.826046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.826026    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-provider-credentials" pod "1532e5a1-3417-487b-95b4-947fb71b20a4" (UID: "1532e5a1-3417-487b-95b4-947fb71b20a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:07.921609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.921572    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:07.921878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.921859    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:07.922183 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.922161    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1532e5a1-3417-487b-95b4-947fb71b20a4-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:07.922360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:07.922342    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btm5k\" (UniqueName: \"kubernetes.io/projected/1532e5a1-3417-487b-95b4-947fb71b20a4-kube-api-access-btm5k\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:08.179976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.179925    4993 generic.go:356] "Generic (PLEG): container finished" podID="77c140cb-a654-47db-b51e-a88e9455d374" containerID="47993b406feb4667333cd8d96f3d77be466d29ff5fbf195525d016db0414255d" exitCode=60
Oct 30 13:30:08.180325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.180294    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" event={"ID":"77c140cb-a654-47db-b51e-a88e9455d374","Type":"ContainerDied","Data":"47993b406feb4667333cd8d96f3d77be466d29ff5fbf195525d016db0414255d"}
Oct 30 13:30:08.180726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.180549    4993 scope.go:122] "RemoveContainer" containerID="1c7c3e596109991b69d6ec4b487879d400e27b5e4944cb7528c6a13514cf20ef"
Oct 30 13:30:08.181994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.181498    4993 scope.go:122] "RemoveContainer" containerID="47993b406feb4667333cd8d96f3d77be466d29ff5fbf195525d016db0414255d"
Oct 30 13:30:08.182954 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.182740    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj_odh-ai-gateway-infra(77c140cb-a654-47db-b51e-a88e9455d374)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" podUID="77c140cb-a654-47db-b51e-a88e9455d374"
Oct 30 13:30:08.240246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.232722    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" event={"ID":"465dbcb3-1211-4b3e-840c-19d89a3e119e","Type":"ContainerStarted","Data":"87f86ca27d0511fc83b756efc4bac9855d209a786becbaac0190962570e22472"}
Oct 30 13:30:08.240246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.238846    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" event={"ID":"45be2541-5839-4419-afe3-3a747093066a","Type":"ContainerStarted","Data":"9e7d915ff4f45a766450130afb0bad219c60973e9be2cbcf342e7954772dcc2c"}
Oct 30 13:30:08.247357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.247025    4993 scope.go:122] "RemoveContainer" containerID="a879d6c08a11f46d26a8acce11d9b140f611475d7faa6811157600b31fd51bee"
Oct 30 13:30:08.252844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.251404    4993 generic.go:356] "Generic (PLEG): container finished" podID="6caff419-28cc-4098-a183-45af8e982db0" containerID="489ad9e6d6ab10d35f2c16bac0061559101a45ab4bc26fff682a723d1a86f1bc" exitCode=60
Oct 30 13:30:08.253154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.251442    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" event={"ID":"6caff419-28cc-4098-a183-45af8e982db0","Type":"ContainerDied","Data":"489ad9e6d6ab10d35f2c16bac0061559101a45ab4bc26fff682a723d1a86f1bc"}
Oct 30 13:30:08.253521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.253503    4993 scope.go:122] "RemoveContainer" containerID="489ad9e6d6ab10d35f2c16bac0061559101a45ab4bc26fff682a723d1a86f1bc"
Oct 30 13:30:08.258013 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.255469    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx_odh-ai-gateway-infra(6caff419-28cc-4098-a183-45af8e982db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" podUID="6caff419-28cc-4098-a183-45af8e982db0"
Oct 30 13:30:08.267608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.267582    4993 scope.go:122] "RemoveContainer" containerID="f32e2eaaa623b3d7fedcbd2a04038ef6e519fa3584d9ab15bf8f2b9b6e76a76c"
Oct 30 13:30:08.272858 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.272827    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w_odh-ai-gateway-infra(dd6514e0-bbbd-4933-836f-fc32631577ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad"
Oct 30 13:30:08.281216 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.281171    4993 scope.go:122] "RemoveContainer" containerID="8af234c59554a1c9d00d3452d0418dbf245da22fba5a021b060ae919cc22e6cc"
Oct 30 13:30:08.281690 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.281525    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw_odh-ai-gateway-infra(5af816ef-1abb-4490-a7e0-4d1901c27223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223"
Oct 30 13:30:08.284103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.284078    4993 generic.go:356] "Generic (PLEG): container finished" podID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerID="509fbd9b9f6dab9c77da83c7cbc4d33c42bdf1917bf8fd18ecd194a4230dac86" exitCode=60
Oct 30 13:30:08.284496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.284472    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" event={"ID":"bdcaa358-f46d-4b45-97bb-3e617acd4ed8","Type":"ContainerDied","Data":"509fbd9b9f6dab9c77da83c7cbc4d33c42bdf1917bf8fd18ecd194a4230dac86"}
Oct 30 13:30:08.285079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.285037    4993 scope.go:122] "RemoveContainer" containerID="509fbd9b9f6dab9c77da83c7cbc4d33c42bdf1917bf8fd18ecd194a4230dac86"
Oct 30 13:30:08.285576 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.285550    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4_odh-ai-gateway-infra(bdcaa358-f46d-4b45-97bb-3e617acd4ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8"
Oct 30 13:30:08.288854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.288562    4993 scope.go:122] "RemoveContainer" containerID="d4abdc14531fb0b28898dac5902e55ff844abfb1ccf73895817b4d8a50fe1cc4"
Oct 30 13:30:08.288951 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.288854    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6_odh-ai-gateway-infra(4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3"
Oct 30 13:30:08.291240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.291181    4993 scope.go:122] "RemoveContainer" containerID="b2bba50eca92c9ef9bc8e332bc6e3d1f8f4d5a61a199ee803c073b84011071e8"
Oct 30 13:30:08.291529 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.291502    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7_odh-ai-gateway-infra(78a20603-e044-4279-a9e8-7c2c7d030a44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44"
Oct 30 13:30:08.293657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.293619    4993 generic.go:356] "Generic (PLEG): container finished" podID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerID="10929dc888e829392456480bfa714cd46e2726c0f42665b578f38f7ce9bd84ed" exitCode=60
Oct 30 13:30:08.293766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.293698    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" event={"ID":"ce27ddf5-ef65-427e-8793-bfbc3015ddbe","Type":"ContainerDied","Data":"10929dc888e829392456480bfa714cd46e2726c0f42665b578f38f7ce9bd84ed"}
Oct 30 13:30:08.295311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.295290    4993 scope.go:122] "RemoveContainer" containerID="10929dc888e829392456480bfa714cd46e2726c0f42665b578f38f7ce9bd84ed"
Oct 30 13:30:08.295761 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.295735    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9_odh-ai-gateway-infra(ce27ddf5-ef65-427e-8793-bfbc3015ddbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe"
Oct 30 13:30:08.296485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.296456    4993 generic.go:356] "Generic (PLEG): container finished" podID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerID="452e1784c0141dc5f1ed4af87bcdd9f75e94f807e429f9d1df6c32d3484978fc" exitCode=60
Oct 30 13:30:08.296590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.296538    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" event={"ID":"d14cbc6e-1967-47a3-be75-b2251ecc25ce","Type":"ContainerDied","Data":"452e1784c0141dc5f1ed4af87bcdd9f75e94f807e429f9d1df6c32d3484978fc"}
Oct 30 13:30:08.296836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.296815    4993 scope.go:122] "RemoveContainer" containerID="452e1784c0141dc5f1ed4af87bcdd9f75e94f807e429f9d1df6c32d3484978fc"
Oct 30 13:30:08.297129 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.297105    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5_odh-ai-gateway-infra(d14cbc6e-1967-47a3-be75-b2251ecc25ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce"
Oct 30 13:30:08.299612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.299586    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" event={"ID":"86002868-e959-4e93-9a73-a6ca9695a40b","Type":"ContainerStarted","Data":"db08c9f657f5d577a0b9ae9662e1fbf5495f246754e3daf3bd02cc5caebd452a"}
Oct 30 13:30:08.299901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.299878    4993 scope.go:122] "RemoveContainer" containerID="db08c9f657f5d577a0b9ae9662e1fbf5495f246754e3daf3bd02cc5caebd452a"
Oct 30 13:30:08.300169 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.300141    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp_odh-ai-gateway-infra(86002868-e959-4e93-9a73-a6ca9695a40b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" podUID="86002868-e959-4e93-9a73-a6ca9695a40b"
Oct 30 13:30:08.301662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.301634    4993 generic.go:356] "Generic (PLEG): container finished" podID="1532e5a1-3417-487b-95b4-947fb71b20a4" containerID="4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d" exitCode=0
Oct 30 13:30:08.301978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.301672    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"
Oct 30 13:30:08.301978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.301701    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6" event={"ID":"1532e5a1-3417-487b-95b4-947fb71b20a4","Type":"ContainerDied","Data":"4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d"}
Oct 30 13:30:08.301978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.301752    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6" event={"ID":"1532e5a1-3417-487b-95b4-947fb71b20a4","Type":"ContainerDied","Data":"a76af8b2d46ad035edab00737f255dc029ba21ec825dc2257bfc901dc1226255"}
Oct 30 13:30:08.304378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.304358    4993 generic.go:356] "Generic (PLEG): container finished" podID="75bc6576-0613-4b84-a683-ed8afd915111" containerID="4bc8d496751e391bd8c8206061ae98016f8565a61af94d38ef53ad67f88206b1" exitCode=60
Oct 30 13:30:08.304539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.304428    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" event={"ID":"75bc6576-0613-4b84-a683-ed8afd915111","Type":"ContainerDied","Data":"4bc8d496751e391bd8c8206061ae98016f8565a61af94d38ef53ad67f88206b1"}
Oct 30 13:30:08.304766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.304746    4993 scope.go:122] "RemoveContainer" containerID="4bc8d496751e391bd8c8206061ae98016f8565a61af94d38ef53ad67f88206b1"
Oct 30 13:30:08.305041 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.305018    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd_odh-ai-gateway-infra(75bc6576-0613-4b84-a683-ed8afd915111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" podUID="75bc6576-0613-4b84-a683-ed8afd915111"
Oct 30 13:30:08.308144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.308120    4993 generic.go:356] "Generic (PLEG): container finished" podID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerID="a903cce12038a206e424496c9acc620c2f056e5ee5475185e0984a45a630ee73" exitCode=60
Oct 30 13:30:08.308281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.308255    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" event={"ID":"6cb2c2fb-83d5-4af2-b030-5368808f2c82","Type":"ContainerDied","Data":"a903cce12038a206e424496c9acc620c2f056e5ee5475185e0984a45a630ee73"}
Oct 30 13:30:08.308852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.308837    4993 scope.go:122] "RemoveContainer" containerID="a903cce12038a206e424496c9acc620c2f056e5ee5475185e0984a45a630ee73"
Oct 30 13:30:08.309246 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.309199    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx_odh-ai-gateway-infra(6cb2c2fb-83d5-4af2-b030-5368808f2c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82"
Oct 30 13:30:08.310487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.310465    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" event={"ID":"187f5533-755c-4fda-b817-7e1d2b97defc","Type":"ContainerStarted","Data":"acb995bd54b9af5791b8d8e25689cd4b4344c5cfe44166c0437214ea1507ff93"}
Oct 30 13:30:08.310749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.310733    4993 scope.go:122] "RemoveContainer" containerID="acb995bd54b9af5791b8d8e25689cd4b4344c5cfe44166c0437214ea1507ff93"
Oct 30 13:30:08.310948 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.310920    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4_odh-ai-gateway-infra(187f5533-755c-4fda-b817-7e1d2b97defc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" podUID="187f5533-755c-4fda-b817-7e1d2b97defc"
Oct 30 13:30:08.312646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.312627    4993 scope.go:122] "RemoveContainer" containerID="d976d98ea937191126d788b4401c937179fbd322e92337e5bf9aef35ddcfe2a9"
Oct 30 13:30:08.312834 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.312815    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4_odh-ai-gateway-infra(c18ca0c2-40ad-4793-982f-176d4efbf373)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373"
Oct 30 13:30:08.314548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.314532    4993 scope.go:122] "RemoveContainer" containerID="8ca83df01c00eac4739cddc6fca5aab610fcfaf5e526c99782fc599b4d4fa3f7"
Oct 30 13:30:08.314850 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.314829    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6_odh-ai-gateway-infra(b0ec16c9-b636-4241-bca4-6bcaf05e9f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11"
Oct 30 13:30:08.374869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.374839    4993 scope.go:122] "RemoveContainer" containerID="950959c1dd3057f47478af38f2758d56e041b99d22650e66c04aa88fa952929a"
Oct 30 13:30:08.395842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.395813    4993 scope.go:122] "RemoveContainer" containerID="5eaefac224df5dd13919bf6cad62b1f1a417ecfa61eb29e653b90e9d4c8d2287"
Oct 30 13:30:08.419300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.419124    4993 scope.go:122] "RemoveContainer" containerID="55e3800914288f97009f2cf172d1fe8fa6534d003373d0671b86dc2cfdda2c95"
Oct 30 13:30:08.434945 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.434902    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1532e5a1_3417_487b_95b4_947fb71b20a4.slice/crio-a76af8b2d46ad035edab00737f255dc029ba21ec825dc2257bfc901dc1226255\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1532e5a1_3417_487b_95b4_947fb71b20a4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:30:08.439394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.439256    4993 scope.go:122] "RemoveContainer" containerID="4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d"
Oct 30 13:30:08.453597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.453571    4993 scope.go:122] "RemoveContainer" containerID="4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d"
Oct 30 13:30:08.454025 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.453990    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d\": container with ID starting with 4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d not found: ID does not exist" containerID="4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d"
Oct 30 13:30:08.454175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.454035    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d"} err="failed to get container status \"4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d\": rpc error: code = NotFound desc = could not find container \"4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d\": container with ID starting with 4f0d574fa1d9656644178cfce0dab1ce557a0e6e18e79b0ea63adbb261cb609d not found: ID does not exist"
Oct 30 13:30:08.454175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.454061    4993 scope.go:122] "RemoveContainer" containerID="48e40ade278497374732f140fcf0e5aeb0bf119fd8de7b50c992c724d382dc27"
Oct 30 13:30:08.478793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.478730    4993 scope.go:122] "RemoveContainer" containerID="62cda6248d0e16ce5dabd01fbe97cd1ddba44777e9945db361615372c654f096"
Oct 30 13:30:08.582157 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:08.582117    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:30:08.871226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.870253    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"]
Oct 30 13:30:08.902671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:08.900499    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-8dd58k6b6"]
Oct 30 13:30:09.356946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.356915    4993 scope.go:122] "RemoveContainer" containerID="10929dc888e829392456480bfa714cd46e2726c0f42665b578f38f7ce9bd84ed"
Oct 30 13:30:09.357594 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.357559    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9_odh-ai-gateway-infra(ce27ddf5-ef65-427e-8793-bfbc3015ddbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe"
Oct 30 13:30:09.364916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.364891    4993 scope.go:122] "RemoveContainer" containerID="452e1784c0141dc5f1ed4af87bcdd9f75e94f807e429f9d1df6c32d3484978fc"
Oct 30 13:30:09.365544 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.365516    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5_odh-ai-gateway-infra(d14cbc6e-1967-47a3-be75-b2251ecc25ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce"
Oct 30 13:30:09.368998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.368971    4993 generic.go:356] "Generic (PLEG): container finished" podID="86002868-e959-4e93-9a73-a6ca9695a40b" containerID="db08c9f657f5d577a0b9ae9662e1fbf5495f246754e3daf3bd02cc5caebd452a" exitCode=60
Oct 30 13:30:09.369287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.369263    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" event={"ID":"86002868-e959-4e93-9a73-a6ca9695a40b","Type":"ContainerDied","Data":"db08c9f657f5d577a0b9ae9662e1fbf5495f246754e3daf3bd02cc5caebd452a"}
Oct 30 13:30:09.369456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.369438    4993 scope.go:122] "RemoveContainer" containerID="e28aa5ef99394a14358e5cb43b34861dc374feaf5224d7d705c725e5a04d02c9"
Oct 30 13:30:09.371606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.371588    4993 scope.go:122] "RemoveContainer" containerID="db08c9f657f5d577a0b9ae9662e1fbf5495f246754e3daf3bd02cc5caebd452a"
Oct 30 13:30:09.376647 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.375047    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp_odh-ai-gateway-infra(86002868-e959-4e93-9a73-a6ca9695a40b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" podUID="86002868-e959-4e93-9a73-a6ca9695a40b"
Oct 30 13:30:09.386604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.386057    4993 scope.go:122] "RemoveContainer" containerID="4bc8d496751e391bd8c8206061ae98016f8565a61af94d38ef53ad67f88206b1"
Oct 30 13:30:09.386604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.386155    4993 scope.go:122] "RemoveContainer" containerID="a903cce12038a206e424496c9acc620c2f056e5ee5475185e0984a45a630ee73"
Oct 30 13:30:09.386604 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.386463    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd_odh-ai-gateway-infra(75bc6576-0613-4b84-a683-ed8afd915111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" podUID="75bc6576-0613-4b84-a683-ed8afd915111"
Oct 30 13:30:09.386604 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.386491    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx_odh-ai-gateway-infra(6cb2c2fb-83d5-4af2-b030-5368808f2c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82"
Oct 30 13:30:09.391253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.390162    4993 generic.go:356] "Generic (PLEG): container finished" podID="187f5533-755c-4fda-b817-7e1d2b97defc" containerID="acb995bd54b9af5791b8d8e25689cd4b4344c5cfe44166c0437214ea1507ff93" exitCode=60
Oct 30 13:30:09.391253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.390333    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" event={"ID":"187f5533-755c-4fda-b817-7e1d2b97defc","Type":"ContainerDied","Data":"acb995bd54b9af5791b8d8e25689cd4b4344c5cfe44166c0437214ea1507ff93"}
Oct 30 13:30:09.392854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.392487    4993 scope.go:122] "RemoveContainer" containerID="acb995bd54b9af5791b8d8e25689cd4b4344c5cfe44166c0437214ea1507ff93"
Oct 30 13:30:09.392854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.392688    4993 scope.go:122] "RemoveContainer" containerID="47993b406feb4667333cd8d96f3d77be466d29ff5fbf195525d016db0414255d"
Oct 30 13:30:09.392854 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.392777    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4_odh-ai-gateway-infra(187f5533-755c-4fda-b817-7e1d2b97defc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" podUID="187f5533-755c-4fda-b817-7e1d2b97defc"
Oct 30 13:30:09.393077 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.392949    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj_odh-ai-gateway-infra(77c140cb-a654-47db-b51e-a88e9455d374)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" podUID="77c140cb-a654-47db-b51e-a88e9455d374"
Oct 30 13:30:09.395721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.395693    4993 generic.go:356] "Generic (PLEG): container finished" podID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerID="87f86ca27d0511fc83b756efc4bac9855d209a786becbaac0190962570e22472" exitCode=60
Oct 30 13:30:09.395825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.395771    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" event={"ID":"465dbcb3-1211-4b3e-840c-19d89a3e119e","Type":"ContainerDied","Data":"87f86ca27d0511fc83b756efc4bac9855d209a786becbaac0190962570e22472"}
Oct 30 13:30:09.396135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.396113    4993 scope.go:122] "RemoveContainer" containerID="87f86ca27d0511fc83b756efc4bac9855d209a786becbaac0190962570e22472"
Oct 30 13:30:09.396835 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.396804    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5_odh-ai-gateway-infra(465dbcb3-1211-4b3e-840c-19d89a3e119e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e"
Oct 30 13:30:09.397887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.397864    4993 scope.go:122] "RemoveContainer" containerID="2d8f967aea32b397b20056df26e464f4f94b29f20ae49407707e266c6d2c0923"
Oct 30 13:30:09.401058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.401037    4993 generic.go:356] "Generic (PLEG): container finished" podID="45be2541-5839-4419-afe3-3a747093066a" containerID="9e7d915ff4f45a766450130afb0bad219c60973e9be2cbcf342e7954772dcc2c" exitCode=60
Oct 30 13:30:09.401289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.401122    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" event={"ID":"45be2541-5839-4419-afe3-3a747093066a","Type":"ContainerDied","Data":"9e7d915ff4f45a766450130afb0bad219c60973e9be2cbcf342e7954772dcc2c"}
Oct 30 13:30:09.401434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.401413    4993 scope.go:122] "RemoveContainer" containerID="9e7d915ff4f45a766450130afb0bad219c60973e9be2cbcf342e7954772dcc2c"
Oct 30 13:30:09.404534 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.401890    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz_odh-ai-gateway-infra(45be2541-5839-4419-afe3-3a747093066a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" podUID="45be2541-5839-4419-afe3-3a747093066a"
Oct 30 13:30:09.404917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.404894    4993 scope.go:122] "RemoveContainer" containerID="489ad9e6d6ab10d35f2c16bac0061559101a45ab4bc26fff682a723d1a86f1bc"
Oct 30 13:30:09.405170 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.405145    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx_odh-ai-gateway-infra(6caff419-28cc-4098-a183-45af8e982db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" podUID="6caff419-28cc-4098-a183-45af8e982db0"
Oct 30 13:30:09.409000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.408690    4993 scope.go:122] "RemoveContainer" containerID="509fbd9b9f6dab9c77da83c7cbc4d33c42bdf1917bf8fd18ecd194a4230dac86"
Oct 30 13:30:09.409000 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:09.408952    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4_odh-ai-gateway-infra(bdcaa358-f46d-4b45-97bb-3e617acd4ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8"
Oct 30 13:30:09.418128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.418106    4993 scope.go:122] "RemoveContainer" containerID="231e5e7dc9207b841003a9b56a06c126fee10b8f72496ba59d6664cd076c09a3"
Oct 30 13:30:09.436156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.436125    4993 scope.go:122] "RemoveContainer" containerID="eb3c51cbcb2af3dec144e1df54e036a0a957ff006b308a1e9b220329ff1a3a2b"
Oct 30 13:30:09.516301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.515181    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:30:09.516301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.515264    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:09.702009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.701930    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1532e5a1-3417-487b-95b4-947fb71b20a4" path="/var/lib/kubelet/pods/1532e5a1-3417-487b-95b4-947fb71b20a4/volumes"
Oct 30 13:30:09.735971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.735936    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:30:09.736315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.736273    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:30:09.737053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:09.737030    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:30:10.422427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:10.422396    4993 scope.go:122] "RemoveContainer" containerID="acb995bd54b9af5791b8d8e25689cd4b4344c5cfe44166c0437214ea1507ff93"
Oct 30 13:30:10.422807 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:10.422782    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4_odh-ai-gateway-infra(187f5533-755c-4fda-b817-7e1d2b97defc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" podUID="187f5533-755c-4fda-b817-7e1d2b97defc"
Oct 30 13:30:10.424327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:10.424301    4993 scope.go:122] "RemoveContainer" containerID="87f86ca27d0511fc83b756efc4bac9855d209a786becbaac0190962570e22472"
Oct 30 13:30:10.424644 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:10.424619    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5_odh-ai-gateway-infra(465dbcb3-1211-4b3e-840c-19d89a3e119e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e"
Oct 30 13:30:10.425976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:10.425958    4993 scope.go:122] "RemoveContainer" containerID="9e7d915ff4f45a766450130afb0bad219c60973e9be2cbcf342e7954772dcc2c"
Oct 30 13:30:10.426274 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:10.426239    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz_odh-ai-gateway-infra(45be2541-5839-4419-afe3-3a747093066a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" podUID="45be2541-5839-4419-afe3-3a747093066a"
Oct 30 13:30:10.427656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:10.427638    4993 scope.go:122] "RemoveContainer" containerID="db08c9f657f5d577a0b9ae9662e1fbf5495f246754e3daf3bd02cc5caebd452a"
Oct 30 13:30:10.427925 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:10.427904    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp_odh-ai-gateway-infra(86002868-e959-4e93-9a73-a6ca9695a40b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" podUID="86002868-e959-4e93-9a73-a6ca9695a40b"
Oct 30 13:30:16.685323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:16.685284    4993 scope.go:122] "RemoveContainer" containerID="d976d98ea937191126d788b4401c937179fbd322e92337e5bf9aef35ddcfe2a9"
Oct 30 13:30:17.495945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:17.495904    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" event={"ID":"c18ca0c2-40ad-4793-982f-176d4efbf373","Type":"ContainerStarted","Data":"2e7faa367606e5321fa1ddeeb41b939caef3c090cbc4eb405c08c9dd5b3cc90f"}
Oct 30 13:30:17.693552 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:17.693521    4993 scope.go:122] "RemoveContainer" containerID="489ad9e6d6ab10d35f2c16bac0061559101a45ab4bc26fff682a723d1a86f1bc"
Oct 30 13:30:17.694084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:17.693673    4993 scope.go:122] "RemoveContainer" containerID="8ca83df01c00eac4739cddc6fca5aab610fcfaf5e526c99782fc599b4d4fa3f7"
Oct 30 13:30:17.694084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:17.693744    4993 scope.go:122] "RemoveContainer" containerID="d4abdc14531fb0b28898dac5902e55ff844abfb1ccf73895817b4d8a50fe1cc4"
Oct 30 13:30:18.509280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.509240    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" event={"ID":"6caff419-28cc-4098-a183-45af8e982db0","Type":"ContainerStarted","Data":"0388c39a026f75e5d9347d12b3ac665cbf467c31c22aa717e91d8a823acc2ac2"}
Oct 30 13:30:18.511439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.511403    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" event={"ID":"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3","Type":"ContainerStarted","Data":"8cf30e02550ed372422980413bb95032555c2d7b5de2612b0e9cc1c599fd967a"}
Oct 30 13:30:18.513644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.513620    4993 generic.go:356] "Generic (PLEG): container finished" podID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerID="2e7faa367606e5321fa1ddeeb41b939caef3c090cbc4eb405c08c9dd5b3cc90f" exitCode=60
Oct 30 13:30:18.513746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.513688    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" event={"ID":"c18ca0c2-40ad-4793-982f-176d4efbf373","Type":"ContainerDied","Data":"2e7faa367606e5321fa1ddeeb41b939caef3c090cbc4eb405c08c9dd5b3cc90f"}
Oct 30 13:30:18.513805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.513748    4993 scope.go:122] "RemoveContainer" containerID="d976d98ea937191126d788b4401c937179fbd322e92337e5bf9aef35ddcfe2a9"
Oct 30 13:30:18.514069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.514053    4993 scope.go:122] "RemoveContainer" containerID="2e7faa367606e5321fa1ddeeb41b939caef3c090cbc4eb405c08c9dd5b3cc90f"
Oct 30 13:30:18.514419 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:18.514391    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4_odh-ai-gateway-infra(c18ca0c2-40ad-4793-982f-176d4efbf373)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373"
Oct 30 13:30:18.515851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.515829    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" event={"ID":"b0ec16c9-b636-4241-bca4-6bcaf05e9f11","Type":"ContainerStarted","Data":"6989755cdebbb038fcf184f548917ecc86fb8b800b95819fb63b5281102bf6f5"}
Oct 30 13:30:18.554897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.554859    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"]
Oct 30 13:30:18.684781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.684752    4993 scope.go:122] "RemoveContainer" containerID="b2bba50eca92c9ef9bc8e332bc6e3d1f8f4d5a61a199ee803c073b84011071e8"
Oct 30 13:30:18.684961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:18.684825    4993 scope.go:122] "RemoveContainer" containerID="8af234c59554a1c9d00d3452d0418dbf245da22fba5a021b060ae919cc22e6cc"
Oct 30 13:30:19.539527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.539494    4993 generic.go:356] "Generic (PLEG): container finished" podID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerID="6989755cdebbb038fcf184f548917ecc86fb8b800b95819fb63b5281102bf6f5" exitCode=60
Oct 30 13:30:19.540041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.539573    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" event={"ID":"b0ec16c9-b636-4241-bca4-6bcaf05e9f11","Type":"ContainerDied","Data":"6989755cdebbb038fcf184f548917ecc86fb8b800b95819fb63b5281102bf6f5"}
Oct 30 13:30:19.540041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.539667    4993 scope.go:122] "RemoveContainer" containerID="8ca83df01c00eac4739cddc6fca5aab610fcfaf5e526c99782fc599b4d4fa3f7"
Oct 30 13:30:19.540041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.540028    4993 scope.go:122] "RemoveContainer" containerID="6989755cdebbb038fcf184f548917ecc86fb8b800b95819fb63b5281102bf6f5"
Oct 30 13:30:19.540443 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:19.540420    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6_odh-ai-gateway-infra(b0ec16c9-b636-4241-bca4-6bcaf05e9f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11"
Oct 30 13:30:19.543047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.543019    4993 generic.go:356] "Generic (PLEG): container finished" podID="6caff419-28cc-4098-a183-45af8e982db0" containerID="0388c39a026f75e5d9347d12b3ac665cbf467c31c22aa717e91d8a823acc2ac2" exitCode=60
Oct 30 13:30:19.543159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.543089    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" event={"ID":"6caff419-28cc-4098-a183-45af8e982db0","Type":"ContainerDied","Data":"0388c39a026f75e5d9347d12b3ac665cbf467c31c22aa717e91d8a823acc2ac2"}
Oct 30 13:30:19.543472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.543454    4993 scope.go:122] "RemoveContainer" containerID="0388c39a026f75e5d9347d12b3ac665cbf467c31c22aa717e91d8a823acc2ac2"
Oct 30 13:30:19.543803 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:19.543778    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx_odh-ai-gateway-infra(6caff419-28cc-4098-a183-45af8e982db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" podUID="6caff419-28cc-4098-a183-45af8e982db0"
Oct 30 13:30:19.545558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.545532    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" event={"ID":"5af816ef-1abb-4490-a7e0-4d1901c27223","Type":"ContainerStarted","Data":"3de58ef7c9a81b0ee883eeaf6f057fd7dfbb3ef3c46163efb70d443654e74cff"}
Oct 30 13:30:19.547883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.547853    4993 generic.go:356] "Generic (PLEG): container finished" podID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerID="8cf30e02550ed372422980413bb95032555c2d7b5de2612b0e9cc1c599fd967a" exitCode=60
Oct 30 13:30:19.547964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.547922    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" event={"ID":"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3","Type":"ContainerDied","Data":"8cf30e02550ed372422980413bb95032555c2d7b5de2612b0e9cc1c599fd967a"}
Oct 30 13:30:19.548347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.548311    4993 scope.go:122] "RemoveContainer" containerID="8cf30e02550ed372422980413bb95032555c2d7b5de2612b0e9cc1c599fd967a"
Oct 30 13:30:19.548672 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:19.548647    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6_odh-ai-gateway-infra(4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3"
Oct 30 13:30:19.550261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.550241    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" event={"ID":"78a20603-e044-4279-a9e8-7c2c7d030a44","Type":"ContainerStarted","Data":"e50e76da38918f3373b30cdf70b014516ac4daf96024016ada1e33d5980f7ce1"}
Oct 30 13:30:19.556746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.556722    4993 scope.go:122] "RemoveContainer" containerID="489ad9e6d6ab10d35f2c16bac0061559101a45ab4bc26fff682a723d1a86f1bc"
Oct 30 13:30:19.570280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.570252    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"]
Oct 30 13:30:19.576073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.576053    4993 scope.go:122] "RemoveContainer" containerID="d4abdc14531fb0b28898dac5902e55ff844abfb1ccf73895817b4d8a50fe1cc4"
Oct 30 13:30:19.672251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.671528    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"]
Oct 30 13:30:19.694779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.694431    4993 scope.go:122] "RemoveContainer" containerID="a903cce12038a206e424496c9acc620c2f056e5ee5475185e0984a45a630ee73"
Oct 30 13:30:19.695622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.695603    4993 scope.go:122] "RemoveContainer" containerID="509fbd9b9f6dab9c77da83c7cbc4d33c42bdf1917bf8fd18ecd194a4230dac86"
Oct 30 13:30:19.696715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.696324    4993 scope.go:122] "RemoveContainer" containerID="87f86ca27d0511fc83b756efc4bac9855d209a786becbaac0190962570e22472"
Oct 30 13:30:19.696715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.696460    4993 scope.go:122] "RemoveContainer" containerID="47993b406feb4667333cd8d96f3d77be466d29ff5fbf195525d016db0414255d"
Oct 30 13:30:19.696715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.696551    4993 scope.go:122] "RemoveContainer" containerID="a879d6c08a11f46d26a8acce11d9b140f611475d7faa6811157600b31fd51bee"
Oct 30 13:30:19.698046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.698026    4993 scope.go:122] "RemoveContainer" containerID="4bc8d496751e391bd8c8206061ae98016f8565a61af94d38ef53ad67f88206b1"
Oct 30 13:30:19.724402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.724365    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"]
Oct 30 13:30:19.725529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.725495    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:19.825747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.825715    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18ca0c2-40ad-4793-982f-176d4efbf373-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18ca0c2-40ad-4793-982f-176d4efbf373-maas-api-service-ca\") pod \"c18ca0c2-40ad-4793-982f-176d4efbf373\" (UID: \"c18ca0c2-40ad-4793-982f-176d4efbf373\") "
Oct 30 13:30:19.825882 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.825841    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18ca0c2-40ad-4793-982f-176d4efbf373-kube-api-access-cwjhm\" (UniqueName: \"kubernetes.io/projected/c18ca0c2-40ad-4793-982f-176d4efbf373-kube-api-access-cwjhm\") pod \"c18ca0c2-40ad-4793-982f-176d4efbf373\" (UID: \"c18ca0c2-40ad-4793-982f-176d4efbf373\") "
Oct 30 13:30:19.827396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.827366    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18ca0c2-40ad-4793-982f-176d4efbf373-maas-api-service-ca" pod "c18ca0c2-40ad-4793-982f-176d4efbf373" (UID: "c18ca0c2-40ad-4793-982f-176d4efbf373"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:19.829868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.829836    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18ca0c2-40ad-4793-982f-176d4efbf373-kube-api-access-cwjhm" pod "c18ca0c2-40ad-4793-982f-176d4efbf373" (UID: "c18ca0c2-40ad-4793-982f-176d4efbf373"). InnerVolumeSpecName "kube-api-access-cwjhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:19.928777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.928739    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwjhm\" (UniqueName: \"kubernetes.io/projected/c18ca0c2-40ad-4793-982f-176d4efbf373-kube-api-access-cwjhm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.928777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:19.928778    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18ca0c2-40ad-4793-982f-176d4efbf373-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.564537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.564504    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4" event={"ID":"c18ca0c2-40ad-4793-982f-176d4efbf373","Type":"ContainerDied","Data":"0cf8fc18148140e8d0777df9124e81dd15b6ea3e3fbe9cba0a3ec7dfe1312f34"}
Oct 30 13:30:20.565014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.564577    4993 scope.go:122] "RemoveContainer" containerID="2e7faa367606e5321fa1ddeeb41b939caef3c090cbc4eb405c08c9dd5b3cc90f"
Oct 30 13:30:20.565014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.564514    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"
Oct 30 13:30:20.569705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.569674    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" event={"ID":"77c140cb-a654-47db-b51e-a88e9455d374","Type":"ContainerStarted","Data":"f07b73a7dbb40c2225b45e37b1a3acac5384d949f7f8ddb96712ef58e4b4fa58"}
Oct 30 13:30:20.572829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.572803    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" event={"ID":"465dbcb3-1211-4b3e-840c-19d89a3e119e","Type":"ContainerStarted","Data":"f5fc3a7063c6da63ae650b39b2a31f220b5b39ec842c1cff29c3d14048d76362"}
Oct 30 13:30:20.575277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.575253    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" event={"ID":"dd6514e0-bbbd-4933-836f-fc32631577ad","Type":"ContainerStarted","Data":"c56d933bbd36f203da51217e28016d65021850cbf9c32cdb471a347027d9e32c"}
Oct 30 13:30:20.580562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.580539    4993 generic.go:356] "Generic (PLEG): container finished" podID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerID="3de58ef7c9a81b0ee883eeaf6f057fd7dfbb3ef3c46163efb70d443654e74cff" exitCode=60
Oct 30 13:30:20.580678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.580609    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" event={"ID":"5af816ef-1abb-4490-a7e0-4d1901c27223","Type":"ContainerDied","Data":"3de58ef7c9a81b0ee883eeaf6f057fd7dfbb3ef3c46163efb70d443654e74cff"}
Oct 30 13:30:20.580678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.580662    4993 scope.go:122] "RemoveContainer" containerID="8af234c59554a1c9d00d3452d0418dbf245da22fba5a021b060ae919cc22e6cc"
Oct 30 13:30:20.580997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.580975    4993 scope.go:122] "RemoveContainer" containerID="3de58ef7c9a81b0ee883eeaf6f057fd7dfbb3ef3c46163efb70d443654e74cff"
Oct 30 13:30:20.581700 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:20.581589    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw_odh-ai-gateway-infra(5af816ef-1abb-4490-a7e0-4d1901c27223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223"
Oct 30 13:30:20.584250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.584231    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" event={"ID":"bdcaa358-f46d-4b45-97bb-3e617acd4ed8","Type":"ContainerStarted","Data":"b6437f7f3d63761c561411a648571328949482cc901f0408e3a29705dc372ada"}
Oct 30 13:30:20.590050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.589279    4993 generic.go:356] "Generic (PLEG): container finished" podID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerID="e50e76da38918f3373b30cdf70b014516ac4daf96024016ada1e33d5980f7ce1" exitCode=60
Oct 30 13:30:20.590050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.589469    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" event={"ID":"78a20603-e044-4279-a9e8-7c2c7d030a44","Type":"ContainerDied","Data":"e50e76da38918f3373b30cdf70b014516ac4daf96024016ada1e33d5980f7ce1"}
Oct 30 13:30:20.590050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.589791    4993 scope.go:122] "RemoveContainer" containerID="e50e76da38918f3373b30cdf70b014516ac4daf96024016ada1e33d5980f7ce1"
Oct 30 13:30:20.590262 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:20.590085    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7_odh-ai-gateway-infra(78a20603-e044-4279-a9e8-7c2c7d030a44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44"
Oct 30 13:30:20.592439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.592415    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" event={"ID":"75bc6576-0613-4b84-a683-ed8afd915111","Type":"ContainerStarted","Data":"19ca0e81a84e722c15e00637e02a914d77f11355e63092dd4034c5b05cca4e83"}
Oct 30 13:30:20.596041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.596016    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" event={"ID":"6cb2c2fb-83d5-4af2-b030-5368808f2c82","Type":"ContainerStarted","Data":"bd5fe8ec4fca778de39e4067bc6dfeeab6fcc6b18c23b05eef47c34f86b88e64"}
Oct 30 13:30:20.613855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.610942    4993 scope.go:122] "RemoveContainer" containerID="b2bba50eca92c9ef9bc8e332bc6e3d1f8f4d5a61a199ee803c073b84011071e8"
Oct 30 13:30:20.685014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.684271    4993 scope.go:122] "RemoveContainer" containerID="452e1784c0141dc5f1ed4af87bcdd9f75e94f807e429f9d1df6c32d3484978fc"
Oct 30 13:30:20.702893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.702778    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"]
Oct 30 13:30:20.888908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.888852    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:20.890900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.890796    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"]
Oct 30 13:30:20.908036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.908003    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"]
Oct 30 13:30:20.917359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.916678    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889450-jnlz4"]
Oct 30 13:30:20.982815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.982789    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-maas-api-service-ca\") pod \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\" (UID: \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\") "
Oct 30 13:30:20.982965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.982948    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-kube-api-access-98rch\" (UniqueName: \"kubernetes.io/projected/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-kube-api-access-98rch\") pod \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\" (UID: \"b0ec16c9-b636-4241-bca4-6bcaf05e9f11\") "
Oct 30 13:30:20.983199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.983176    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-maas-api-service-ca" pod "b0ec16c9-b636-4241-bca4-6bcaf05e9f11" (UID: "b0ec16c9-b636-4241-bca4-6bcaf05e9f11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:20.985319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:20.985294    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-kube-api-access-98rch" pod "b0ec16c9-b636-4241-bca4-6bcaf05e9f11" (UID: "b0ec16c9-b636-4241-bca4-6bcaf05e9f11"). InnerVolumeSpecName "kube-api-access-98rch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.084486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.084453    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98rch\" (UniqueName: \"kubernetes.io/projected/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-kube-api-access-98rch\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.084486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.084483    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec16c9-b636-4241-bca4-6bcaf05e9f11-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.190602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.190541    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" podStartSLOduration=21.190521616 podStartE2EDuration="21.190521616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:21.188562992 +0000 UTC m=+3485.664282603" watchObservedRunningTime="2026-10-30 13:30:21.190521616 +0000 UTC m=+3485.666241227"
Oct 30 13:30:21.275835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.275809    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:21.280801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.280777    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:21.287171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.287151    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6caff419-28cc-4098-a183-45af8e982db0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6caff419-28cc-4098-a183-45af8e982db0-maas-api-service-ca\") pod \"6caff419-28cc-4098-a183-45af8e982db0\" (UID: \"6caff419-28cc-4098-a183-45af8e982db0\") "
Oct 30 13:30:21.287280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.287182    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-maas-api-service-ca\") pod \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\" (UID: \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\") "
Oct 30 13:30:21.287352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.287336    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6caff419-28cc-4098-a183-45af8e982db0-kube-api-access-4h8mh\" (UniqueName: \"kubernetes.io/projected/6caff419-28cc-4098-a183-45af8e982db0-kube-api-access-4h8mh\") pod \"6caff419-28cc-4098-a183-45af8e982db0\" (UID: \"6caff419-28cc-4098-a183-45af8e982db0\") "
Oct 30 13:30:21.287395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.287375    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-kube-api-access-972qg\" (UniqueName: \"kubernetes.io/projected/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-kube-api-access-972qg\") pod \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\" (UID: \"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3\") "
Oct 30 13:30:21.287601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.287575    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6caff419-28cc-4098-a183-45af8e982db0-maas-api-service-ca" pod "6caff419-28cc-4098-a183-45af8e982db0" (UID: "6caff419-28cc-4098-a183-45af8e982db0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:21.287674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.287648    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-maas-api-service-ca" pod "4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" (UID: "4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:21.287941 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.287922    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6caff419-28cc-4098-a183-45af8e982db0-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.288038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.287943    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.289603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.289578    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6caff419-28cc-4098-a183-45af8e982db0-kube-api-access-4h8mh" pod "6caff419-28cc-4098-a183-45af8e982db0" (UID: "6caff419-28cc-4098-a183-45af8e982db0"). InnerVolumeSpecName "kube-api-access-4h8mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.289677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.289613    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-kube-api-access-972qg" pod "4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" (UID: "4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3"). InnerVolumeSpecName "kube-api-access-972qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.388895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.388820    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h8mh\" (UniqueName: \"kubernetes.io/projected/6caff419-28cc-4098-a183-45af8e982db0-kube-api-access-4h8mh\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.388895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.388851    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972qg\" (UniqueName: \"kubernetes.io/projected/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3-kube-api-access-972qg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.611366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.611326    4993 generic.go:356] "Generic (PLEG): container finished" podID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerID="f5fc3a7063c6da63ae650b39b2a31f220b5b39ec842c1cff29c3d14048d76362" exitCode=60
Oct 30 13:30:21.611843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.611364    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" event={"ID":"465dbcb3-1211-4b3e-840c-19d89a3e119e","Type":"ContainerDied","Data":"f5fc3a7063c6da63ae650b39b2a31f220b5b39ec842c1cff29c3d14048d76362"}
Oct 30 13:30:21.611843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.611442    4993 scope.go:122] "RemoveContainer" containerID="87f86ca27d0511fc83b756efc4bac9855d209a786becbaac0190962570e22472"
Oct 30 13:30:21.611972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.611843    4993 scope.go:122] "RemoveContainer" containerID="f5fc3a7063c6da63ae650b39b2a31f220b5b39ec842c1cff29c3d14048d76362"
Oct 30 13:30:21.612310 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:21.612279    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5_odh-ai-gateway-infra(465dbcb3-1211-4b3e-840c-19d89a3e119e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e"
Oct 30 13:30:21.614152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.614124    4993 generic.go:356] "Generic (PLEG): container finished" podID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerID="c56d933bbd36f203da51217e28016d65021850cbf9c32cdb471a347027d9e32c" exitCode=60
Oct 30 13:30:21.614300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.614240    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" event={"ID":"dd6514e0-bbbd-4933-836f-fc32631577ad","Type":"ContainerDied","Data":"c56d933bbd36f203da51217e28016d65021850cbf9c32cdb471a347027d9e32c"}
Oct 30 13:30:21.614655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.614633    4993 scope.go:122] "RemoveContainer" containerID="c56d933bbd36f203da51217e28016d65021850cbf9c32cdb471a347027d9e32c"
Oct 30 13:30:21.614989 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:21.614961    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w_odh-ai-gateway-infra(dd6514e0-bbbd-4933-836f-fc32631577ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad"
Oct 30 13:30:21.616326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.616302    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"
Oct 30 13:30:21.616512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.616300    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx" event={"ID":"6caff419-28cc-4098-a183-45af8e982db0","Type":"ContainerDied","Data":"262cc2a9236e769f74b6523d245ad4e970ae0755d46a6c8fcfa896ef9de860a6"}
Oct 30 13:30:21.628546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.627449    4993 generic.go:356] "Generic (PLEG): container finished" podID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerID="b6437f7f3d63761c561411a648571328949482cc901f0408e3a29705dc372ada" exitCode=60
Oct 30 13:30:21.628546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.627568    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" event={"ID":"bdcaa358-f46d-4b45-97bb-3e617acd4ed8","Type":"ContainerDied","Data":"b6437f7f3d63761c561411a648571328949482cc901f0408e3a29705dc372ada"}
Oct 30 13:30:21.628546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.627992    4993 scope.go:122] "RemoveContainer" containerID="b6437f7f3d63761c561411a648571328949482cc901f0408e3a29705dc372ada"
Oct 30 13:30:21.628546 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:21.628432    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4_odh-ai-gateway-infra(bdcaa358-f46d-4b45-97bb-3e617acd4ed8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8"
Oct 30 13:30:21.629431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.629411    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"
Oct 30 13:30:21.629431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.629426    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6" event={"ID":"4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3","Type":"ContainerDied","Data":"8e33c895021e7619a9781f46d36ae5c1ab25bbad69508c323599bc4de803ada4"}
Oct 30 13:30:21.630930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.630912    4993 scope.go:122] "RemoveContainer" containerID="a879d6c08a11f46d26a8acce11d9b140f611475d7faa6811157600b31fd51bee"
Oct 30 13:30:21.634314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.634251    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" event={"ID":"d14cbc6e-1967-47a3-be75-b2251ecc25ce","Type":"ContainerStarted","Data":"03f58b1c6af38a440a1482e7d3d307c09bdbe01ea1a7ac6704a45022dd557c65"}
Oct 30 13:30:21.636641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.636619    4993 generic.go:356] "Generic (PLEG): container finished" podID="75bc6576-0613-4b84-a683-ed8afd915111" containerID="19ca0e81a84e722c15e00637e02a914d77f11355e63092dd4034c5b05cca4e83" exitCode=60
Oct 30 13:30:21.636757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.636701    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" event={"ID":"75bc6576-0613-4b84-a683-ed8afd915111","Type":"ContainerDied","Data":"19ca0e81a84e722c15e00637e02a914d77f11355e63092dd4034c5b05cca4e83"}
Oct 30 13:30:21.637075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.637055    4993 scope.go:122] "RemoveContainer" containerID="19ca0e81a84e722c15e00637e02a914d77f11355e63092dd4034c5b05cca4e83"
Oct 30 13:30:21.637436 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:21.637409    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd_odh-ai-gateway-infra(75bc6576-0613-4b84-a683-ed8afd915111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" podUID="75bc6576-0613-4b84-a683-ed8afd915111"
Oct 30 13:30:21.639578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.639495    4993 generic.go:356] "Generic (PLEG): container finished" podID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerID="bd5fe8ec4fca778de39e4067bc6dfeeab6fcc6b18c23b05eef47c34f86b88e64" exitCode=60
Oct 30 13:30:21.639735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.639636    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" event={"ID":"6cb2c2fb-83d5-4af2-b030-5368808f2c82","Type":"ContainerDied","Data":"bd5fe8ec4fca778de39e4067bc6dfeeab6fcc6b18c23b05eef47c34f86b88e64"}
Oct 30 13:30:21.639944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.639928    4993 scope.go:122] "RemoveContainer" containerID="bd5fe8ec4fca778de39e4067bc6dfeeab6fcc6b18c23b05eef47c34f86b88e64"
Oct 30 13:30:21.640241 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:21.640179    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx_odh-ai-gateway-infra(6cb2c2fb-83d5-4af2-b030-5368808f2c82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82"
Oct 30 13:30:21.643176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.643156    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"
Oct 30 13:30:21.643298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.643235    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6" event={"ID":"b0ec16c9-b636-4241-bca4-6bcaf05e9f11","Type":"ContainerDied","Data":"c2c3eb2e9f3a9f99d710ab65d460c963867f82bb12392e337fdfbdced8d82a82"}
Oct 30 13:30:21.649925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.648907    4993 generic.go:356] "Generic (PLEG): container finished" podID="77c140cb-a654-47db-b51e-a88e9455d374" containerID="f07b73a7dbb40c2225b45e37b1a3acac5384d949f7f8ddb96712ef58e4b4fa58" exitCode=60
Oct 30 13:30:21.649925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.648941    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" event={"ID":"77c140cb-a654-47db-b51e-a88e9455d374","Type":"ContainerDied","Data":"f07b73a7dbb40c2225b45e37b1a3acac5384d949f7f8ddb96712ef58e4b4fa58"}
Oct 30 13:30:21.649925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.649536    4993 scope.go:122] "RemoveContainer" containerID="f07b73a7dbb40c2225b45e37b1a3acac5384d949f7f8ddb96712ef58e4b4fa58"
Oct 30 13:30:21.649925 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:21.649834    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj_odh-ai-gateway-infra(77c140cb-a654-47db-b51e-a88e9455d374)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" podUID="77c140cb-a654-47db-b51e-a88e9455d374"
Oct 30 13:30:21.655317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.655029    4993 scope.go:122] "RemoveContainer" containerID="0388c39a026f75e5d9347d12b3ac665cbf467c31c22aa717e91d8a823acc2ac2"
Oct 30 13:30:21.681227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.681176    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"]
Oct 30 13:30:21.690066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.688381    4993 scope.go:122] "RemoveContainer" containerID="509fbd9b9f6dab9c77da83c7cbc4d33c42bdf1917bf8fd18ecd194a4230dac86"
Oct 30 13:30:21.710321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.705924    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" path="/var/lib/kubelet/pods/c18ca0c2-40ad-4793-982f-176d4efbf373/volumes"
Oct 30 13:30:21.710321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.709035    4993 scope.go:122] "RemoveContainer" containerID="acb995bd54b9af5791b8d8e25689cd4b4344c5cfe44166c0437214ea1507ff93"
Oct 30 13:30:21.710321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.709969    4993 scope.go:122] "RemoveContainer" containerID="10929dc888e829392456480bfa714cd46e2726c0f42665b578f38f7ce9bd84ed"
Oct 30 13:30:21.714475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.714448    4993 scope.go:122] "RemoveContainer" containerID="db08c9f657f5d577a0b9ae9662e1fbf5495f246754e3daf3bd02cc5caebd452a"
Oct 30 13:30:21.766580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.761860    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"]
Oct 30 13:30:21.778970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.770645    4993 scope.go:122] "RemoveContainer" containerID="8cf30e02550ed372422980413bb95032555c2d7b5de2612b0e9cc1c599fd967a"
Oct 30 13:30:21.832955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.832922    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"]
Oct 30 13:30:21.868430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.868404    4993 scope.go:122] "RemoveContainer" containerID="4bc8d496751e391bd8c8206061ae98016f8565a61af94d38ef53ad67f88206b1"
Oct 30 13:30:21.890639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.890569    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:21.893565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.893544    4993 scope.go:122] "RemoveContainer" containerID="a903cce12038a206e424496c9acc620c2f056e5ee5475185e0984a45a630ee73"
Oct 30 13:30:21.896342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.896314    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:21.908774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.908740    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af816ef-1abb-4490-a7e0-4d1901c27223-kube-api-access-xf87f\" (UniqueName: \"kubernetes.io/projected/5af816ef-1abb-4490-a7e0-4d1901c27223-kube-api-access-xf87f\") pod \"5af816ef-1abb-4490-a7e0-4d1901c27223\" (UID: \"5af816ef-1abb-4490-a7e0-4d1901c27223\") "
Oct 30 13:30:21.908939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.908806    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a20603-e044-4279-a9e8-7c2c7d030a44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a20603-e044-4279-a9e8-7c2c7d030a44-maas-api-service-ca\") pod \"78a20603-e044-4279-a9e8-7c2c7d030a44\" (UID: \"78a20603-e044-4279-a9e8-7c2c7d030a44\") "
Oct 30 13:30:21.909048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.909025    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af816ef-1abb-4490-a7e0-4d1901c27223-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af816ef-1abb-4490-a7e0-4d1901c27223-maas-api-service-ca\") pod \"5af816ef-1abb-4490-a7e0-4d1901c27223\" (UID: \"5af816ef-1abb-4490-a7e0-4d1901c27223\") "
Oct 30 13:30:21.909131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.909072    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a20603-e044-4279-a9e8-7c2c7d030a44-kube-api-access-vrzkf\" (UniqueName: \"kubernetes.io/projected/78a20603-e044-4279-a9e8-7c2c7d030a44-kube-api-access-vrzkf\") pod \"78a20603-e044-4279-a9e8-7c2c7d030a44\" (UID: \"78a20603-e044-4279-a9e8-7c2c7d030a44\") "
Oct 30 13:30:21.909328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.909248    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a20603-e044-4279-a9e8-7c2c7d030a44-maas-api-service-ca" pod "78a20603-e044-4279-a9e8-7c2c7d030a44" (UID: "78a20603-e044-4279-a9e8-7c2c7d030a44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:21.909440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.909418    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af816ef-1abb-4490-a7e0-4d1901c27223-maas-api-service-ca" pod "5af816ef-1abb-4490-a7e0-4d1901c27223" (UID: "5af816ef-1abb-4490-a7e0-4d1901c27223"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:21.909784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.909766    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a20603-e044-4279-a9e8-7c2c7d030a44-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.909784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.909784    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af816ef-1abb-4490-a7e0-4d1901c27223-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.910671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.910647    4993 scope.go:122] "RemoveContainer" containerID="6989755cdebbb038fcf184f548917ecc86fb8b800b95819fb63b5281102bf6f5"
Oct 30 13:30:21.911673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.911648    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a20603-e044-4279-a9e8-7c2c7d030a44-kube-api-access-vrzkf" pod "78a20603-e044-4279-a9e8-7c2c7d030a44" (UID: "78a20603-e044-4279-a9e8-7c2c7d030a44"). InnerVolumeSpecName "kube-api-access-vrzkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.911782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.911698    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af816ef-1abb-4490-a7e0-4d1901c27223-kube-api-access-xf87f" pod "5af816ef-1abb-4490-a7e0-4d1901c27223" (UID: "5af816ef-1abb-4490-a7e0-4d1901c27223"). InnerVolumeSpecName "kube-api-access-xf87f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.925262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:21.925233    4993 scope.go:122] "RemoveContainer" containerID="47993b406feb4667333cd8d96f3d77be466d29ff5fbf195525d016db0414255d"
Oct 30 13:30:22.007686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.007648    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"]
Oct 30 13:30:22.010832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.010798    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf87f\" (UniqueName: \"kubernetes.io/projected/5af816ef-1abb-4490-a7e0-4d1901c27223-kube-api-access-xf87f\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.010832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.010831    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrzkf\" (UniqueName: \"kubernetes.io/projected/78a20603-e044-4279-a9e8-7c2c7d030a44-kube-api-access-vrzkf\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.013184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.013153    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889450-jgcl6"]
Oct 30 13:30:22.069596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.069550    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"]
Oct 30 13:30:22.102855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.102666    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"]
Oct 30 13:30:22.113021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.112972    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" podStartSLOduration=22.112957316 podStartE2EDuration="22.112957316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:22.101088496 +0000 UTC m=+3486.576808112" watchObservedRunningTime="2026-10-30 13:30:22.112957316 +0000 UTC m=+3486.588676928"
Oct 30 13:30:22.167282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.167141    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"]
Oct 30 13:30:22.172720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.172687    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889450-s85sx"]
Oct 30 13:30:22.216046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.216004    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"]
Oct 30 13:30:22.358112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.358045    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"]
Oct 30 13:30:22.368521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.366971    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889450-7nxv6"]
Oct 30 13:30:22.669380 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.669335    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" event={"ID":"187f5533-755c-4fda-b817-7e1d2b97defc","Type":"ContainerStarted","Data":"50bc6cacf53a920ece8cbaf6b2c520fe24f0bbcbee2786e99bbf9ff488468ab8"}
Oct 30 13:30:22.686288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.685869    4993 scope.go:122] "RemoveContainer" containerID="9e7d915ff4f45a766450130afb0bad219c60973e9be2cbcf342e7954772dcc2c"
Oct 30 13:30:22.694479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.690402    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw" event={"ID":"5af816ef-1abb-4490-a7e0-4d1901c27223","Type":"ContainerDied","Data":"931d76de34c827c8c77ad4f0e4670eb0569ede2cb11e50a9e345a1317c55258a"}
Oct 30 13:30:22.694479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.690466    4993 scope.go:122] "RemoveContainer" containerID="3de58ef7c9a81b0ee883eeaf6f057fd7dfbb3ef3c46163efb70d443654e74cff"
Oct 30 13:30:22.694479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.690624    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"
Oct 30 13:30:22.726120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.722221    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"
Oct 30 13:30:22.726120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.725271    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7" event={"ID":"78a20603-e044-4279-a9e8-7c2c7d030a44","Type":"ContainerDied","Data":"1f6fd120a2eeb2ba3336ec0431ede20847afee5128bfc6538492c7cf6db704c0"}
Oct 30 13:30:22.738066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.738014    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" event={"ID":"ce27ddf5-ef65-427e-8793-bfbc3015ddbe","Type":"ContainerStarted","Data":"de792354897b2322256e39beb6115f50da3fe1c10ed0ead90dc9f7cd7ac8c1b3"}
Oct 30 13:30:22.744237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.742764    4993 generic.go:356] "Generic (PLEG): container finished" podID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerID="03f58b1c6af38a440a1482e7d3d307c09bdbe01ea1a7ac6704a45022dd557c65" exitCode=60
Oct 30 13:30:22.744237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.742859    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" event={"ID":"d14cbc6e-1967-47a3-be75-b2251ecc25ce","Type":"ContainerDied","Data":"03f58b1c6af38a440a1482e7d3d307c09bdbe01ea1a7ac6704a45022dd557c65"}
Oct 30 13:30:22.744237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.743366    4993 scope.go:122] "RemoveContainer" containerID="03f58b1c6af38a440a1482e7d3d307c09bdbe01ea1a7ac6704a45022dd557c65"
Oct 30 13:30:22.744237 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:22.743762    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5_odh-ai-gateway-infra(d14cbc6e-1967-47a3-be75-b2251ecc25ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce"
Oct 30 13:30:22.750523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.750496    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" event={"ID":"86002868-e959-4e93-9a73-a6ca9695a40b","Type":"ContainerStarted","Data":"4fa38d59de6c621397a608e8c50d14122fa63094003d558f27bea02bfb66b6f1"}
Oct 30 13:30:22.820310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.820184    4993 scope.go:122] "RemoveContainer" containerID="e50e76da38918f3373b30cdf70b014516ac4daf96024016ada1e33d5980f7ce1"
Oct 30 13:30:22.840406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:22.840267    4993 scope.go:122] "RemoveContainer" containerID="452e1784c0141dc5f1ed4af87bcdd9f75e94f807e429f9d1df6c32d3484978fc"
Oct 30 13:30:23.139809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.139778    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"]
Oct 30 13:30:23.162340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.162130    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"]
Oct 30 13:30:23.169508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.168730    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889450-sxgtw"]
Oct 30 13:30:23.221525 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.221448    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:23.226982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.226950    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:23.233238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.233181    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:23.238976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.238952    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:23.249408 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.248318    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"]
Oct 30 13:30:23.252219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.252015    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:23.253124 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253049    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75bc6576-0613-4b84-a683-ed8afd915111-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75bc6576-0613-4b84-a683-ed8afd915111-maas-api-service-ca\") pod \"75bc6576-0613-4b84-a683-ed8afd915111\" (UID: \"75bc6576-0613-4b84-a683-ed8afd915111\") "
Oct 30 13:30:23.253268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253127    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb2c2fb-83d5-4af2-b030-5368808f2c82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c2fb-83d5-4af2-b030-5368808f2c82-maas-api-service-ca\") pod \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\" (UID: \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\") "
Oct 30 13:30:23.253268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253180    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bc6576-0613-4b84-a683-ed8afd915111-kube-api-access-nbztb\" (UniqueName: \"kubernetes.io/projected/75bc6576-0613-4b84-a683-ed8afd915111-kube-api-access-nbztb\") pod \"75bc6576-0613-4b84-a683-ed8afd915111\" (UID: \"75bc6576-0613-4b84-a683-ed8afd915111\") "
Oct 30 13:30:23.253268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253262    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465dbcb3-1211-4b3e-840c-19d89a3e119e-kube-api-access-kkz9j\" (UniqueName: \"kubernetes.io/projected/465dbcb3-1211-4b3e-840c-19d89a3e119e-kube-api-access-kkz9j\") pod \"465dbcb3-1211-4b3e-840c-19d89a3e119e\" (UID: \"465dbcb3-1211-4b3e-840c-19d89a3e119e\") "
Oct 30 13:30:23.254247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253411    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465dbcb3-1211-4b3e-840c-19d89a3e119e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465dbcb3-1211-4b3e-840c-19d89a3e119e-maas-api-service-ca\") pod \"465dbcb3-1211-4b3e-840c-19d89a3e119e\" (UID: \"465dbcb3-1211-4b3e-840c-19d89a3e119e\") "
Oct 30 13:30:23.254247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253514    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6514e0-bbbd-4933-836f-fc32631577ad-kube-api-access-q5wm6\" (UniqueName: \"kubernetes.io/projected/dd6514e0-bbbd-4933-836f-fc32631577ad-kube-api-access-q5wm6\") pod \"dd6514e0-bbbd-4933-836f-fc32631577ad\" (UID: \"dd6514e0-bbbd-4933-836f-fc32631577ad\") "
Oct 30 13:30:23.254247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253600    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb2c2fb-83d5-4af2-b030-5368808f2c82-kube-api-access-8pgrs\" (UniqueName: \"kubernetes.io/projected/6cb2c2fb-83d5-4af2-b030-5368808f2c82-kube-api-access-8pgrs\") pod \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\" (UID: \"6cb2c2fb-83d5-4af2-b030-5368808f2c82\") "
Oct 30 13:30:23.254247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253594    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb2c2fb-83d5-4af2-b030-5368808f2c82-maas-api-service-ca" pod "6cb2c2fb-83d5-4af2-b030-5368808f2c82" (UID: "6cb2c2fb-83d5-4af2-b030-5368808f2c82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:23.254247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253671    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6514e0-bbbd-4933-836f-fc32631577ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6514e0-bbbd-4933-836f-fc32631577ad-maas-api-service-ca\") pod \"dd6514e0-bbbd-4933-836f-fc32631577ad\" (UID: \"dd6514e0-bbbd-4933-836f-fc32631577ad\") "
Oct 30 13:30:23.254247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.253947    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75bc6576-0613-4b84-a683-ed8afd915111-maas-api-service-ca" pod "75bc6576-0613-4b84-a683-ed8afd915111" (UID: "75bc6576-0613-4b84-a683-ed8afd915111"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:23.254574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.254299    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465dbcb3-1211-4b3e-840c-19d89a3e119e-maas-api-service-ca" pod "465dbcb3-1211-4b3e-840c-19d89a3e119e" (UID: "465dbcb3-1211-4b3e-840c-19d89a3e119e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:23.254657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.254633    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75bc6576-0613-4b84-a683-ed8afd915111-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.254709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.254666    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c2fb-83d5-4af2-b030-5368808f2c82-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.254709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.254682    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/465dbcb3-1211-4b3e-840c-19d89a3e119e-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.261100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.260813    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889450-gzql7"]
Oct 30 13:30:23.261327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.261291    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6514e0-bbbd-4933-836f-fc32631577ad-maas-api-service-ca" pod "dd6514e0-bbbd-4933-836f-fc32631577ad" (UID: "dd6514e0-bbbd-4933-836f-fc32631577ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:23.261509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.261485    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc6576-0613-4b84-a683-ed8afd915111-kube-api-access-nbztb" pod "75bc6576-0613-4b84-a683-ed8afd915111" (UID: "75bc6576-0613-4b84-a683-ed8afd915111"). InnerVolumeSpecName "kube-api-access-nbztb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:23.264541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.264364    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:23.268695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.268524    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465dbcb3-1211-4b3e-840c-19d89a3e119e-kube-api-access-kkz9j" pod "465dbcb3-1211-4b3e-840c-19d89a3e119e" (UID: "465dbcb3-1211-4b3e-840c-19d89a3e119e"). InnerVolumeSpecName "kube-api-access-kkz9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:23.268695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.268530    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6514e0-bbbd-4933-836f-fc32631577ad-kube-api-access-q5wm6" pod "dd6514e0-bbbd-4933-836f-fc32631577ad" (UID: "dd6514e0-bbbd-4933-836f-fc32631577ad"). InnerVolumeSpecName "kube-api-access-q5wm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:23.273682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.273651    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb2c2fb-83d5-4af2-b030-5368808f2c82-kube-api-access-8pgrs" pod "6cb2c2fb-83d5-4af2-b030-5368808f2c82" (UID: "6cb2c2fb-83d5-4af2-b030-5368808f2c82"). InnerVolumeSpecName "kube-api-access-8pgrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:23.357495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.357457    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c140cb-a654-47db-b51e-a88e9455d374-kube-api-access-bz52t\" (UniqueName: \"kubernetes.io/projected/77c140cb-a654-47db-b51e-a88e9455d374-kube-api-access-bz52t\") pod \"77c140cb-a654-47db-b51e-a88e9455d374\" (UID: \"77c140cb-a654-47db-b51e-a88e9455d374\") "
Oct 30 13:30:23.357749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.357564    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-kube-api-access-p9fpw\" (UniqueName: \"kubernetes.io/projected/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-kube-api-access-p9fpw\") pod \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\" (UID: \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\") "
Oct 30 13:30:23.357749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.357620    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-maas-api-service-ca\") pod \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\" (UID: \"bdcaa358-f46d-4b45-97bb-3e617acd4ed8\") "
Oct 30 13:30:23.357919 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.357803    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c140cb-a654-47db-b51e-a88e9455d374-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c140cb-a654-47db-b51e-a88e9455d374-maas-api-service-ca\") pod \"77c140cb-a654-47db-b51e-a88e9455d374\" (UID: \"77c140cb-a654-47db-b51e-a88e9455d374\") "
Oct 30 13:30:23.358427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.358401    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbztb\" (UniqueName: \"kubernetes.io/projected/75bc6576-0613-4b84-a683-ed8afd915111-kube-api-access-nbztb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.358427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.358428    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkz9j\" (UniqueName: \"kubernetes.io/projected/465dbcb3-1211-4b3e-840c-19d89a3e119e-kube-api-access-kkz9j\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.358670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.358442    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wm6\" (UniqueName: \"kubernetes.io/projected/dd6514e0-bbbd-4933-836f-fc32631577ad-kube-api-access-q5wm6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.358670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.358457    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgrs\" (UniqueName: \"kubernetes.io/projected/6cb2c2fb-83d5-4af2-b030-5368808f2c82-kube-api-access-8pgrs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.358670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.358470    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6514e0-bbbd-4933-836f-fc32631577ad-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.359333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.359290    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-maas-api-service-ca" pod "bdcaa358-f46d-4b45-97bb-3e617acd4ed8" (UID: "bdcaa358-f46d-4b45-97bb-3e617acd4ed8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:23.359830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.359810    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c140cb-a654-47db-b51e-a88e9455d374-maas-api-service-ca" pod "77c140cb-a654-47db-b51e-a88e9455d374" (UID: "77c140cb-a654-47db-b51e-a88e9455d374"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:23.363779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.363742    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-kube-api-access-p9fpw" pod "bdcaa358-f46d-4b45-97bb-3e617acd4ed8" (UID: "bdcaa358-f46d-4b45-97bb-3e617acd4ed8"). InnerVolumeSpecName "kube-api-access-p9fpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:23.364401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.364338    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c140cb-a654-47db-b51e-a88e9455d374-kube-api-access-bz52t" pod "77c140cb-a654-47db-b51e-a88e9455d374" (UID: "77c140cb-a654-47db-b51e-a88e9455d374"). InnerVolumeSpecName "kube-api-access-bz52t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:23.460248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.460198    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz52t\" (UniqueName: \"kubernetes.io/projected/77c140cb-a654-47db-b51e-a88e9455d374-kube-api-access-bz52t\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.460248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.460247    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9fpw\" (UniqueName: \"kubernetes.io/projected/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-kube-api-access-p9fpw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.460458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.460262    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdcaa358-f46d-4b45-97bb-3e617acd4ed8-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.460458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.460275    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c140cb-a654-47db-b51e-a88e9455d374-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:23.694997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.694956    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" path="/var/lib/kubelet/pods/4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3/volumes"
Oct 30 13:30:23.695539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.695444    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" path="/var/lib/kubelet/pods/5af816ef-1abb-4490-a7e0-4d1901c27223/volumes"
Oct 30 13:30:23.695836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.695819    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6caff419-28cc-4098-a183-45af8e982db0" path="/var/lib/kubelet/pods/6caff419-28cc-4098-a183-45af8e982db0/volumes"
Oct 30 13:30:23.696559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.696538    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" path="/var/lib/kubelet/pods/78a20603-e044-4279-a9e8-7c2c7d030a44/volumes"
Oct 30 13:30:23.696945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.696929    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" path="/var/lib/kubelet/pods/b0ec16c9-b636-4241-bca4-6bcaf05e9f11/volumes"
Oct 30 13:30:23.762529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.762490    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4" event={"ID":"bdcaa358-f46d-4b45-97bb-3e617acd4ed8","Type":"ContainerDied","Data":"152ec0bd597addd059e273f805bdffbbfd483939bcd124a1f253d6d9c4167f78"}
Oct 30 13:30:23.762665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.762566    4993 scope.go:122] "RemoveContainer" containerID="b6437f7f3d63761c561411a648571328949482cc901f0408e3a29705dc372ada"
Oct 30 13:30:23.762704 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.762679    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"
Oct 30 13:30:23.767602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.767572    4993 generic.go:356] "Generic (PLEG): container finished" podID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerID="de792354897b2322256e39beb6115f50da3fe1c10ed0ead90dc9f7cd7ac8c1b3" exitCode=60
Oct 30 13:30:23.767733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.767615    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" event={"ID":"ce27ddf5-ef65-427e-8793-bfbc3015ddbe","Type":"ContainerDied","Data":"de792354897b2322256e39beb6115f50da3fe1c10ed0ead90dc9f7cd7ac8c1b3"}
Oct 30 13:30:23.768056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.768034    4993 scope.go:122] "RemoveContainer" containerID="de792354897b2322256e39beb6115f50da3fe1c10ed0ead90dc9f7cd7ac8c1b3"
Oct 30 13:30:23.768382 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:23.768360    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9_odh-ai-gateway-infra(ce27ddf5-ef65-427e-8793-bfbc3015ddbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe"
Oct 30 13:30:23.771906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.771886    4993 generic.go:356] "Generic (PLEG): container finished" podID="86002868-e959-4e93-9a73-a6ca9695a40b" containerID="4fa38d59de6c621397a608e8c50d14122fa63094003d558f27bea02bfb66b6f1" exitCode=60
Oct 30 13:30:23.772019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.771957    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" event={"ID":"86002868-e959-4e93-9a73-a6ca9695a40b","Type":"ContainerDied","Data":"4fa38d59de6c621397a608e8c50d14122fa63094003d558f27bea02bfb66b6f1"}
Oct 30 13:30:23.772390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.772362    4993 scope.go:122] "RemoveContainer" containerID="4fa38d59de6c621397a608e8c50d14122fa63094003d558f27bea02bfb66b6f1"
Oct 30 13:30:23.772642 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:23.772621    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp_odh-ai-gateway-infra(86002868-e959-4e93-9a73-a6ca9695a40b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" podUID="86002868-e959-4e93-9a73-a6ca9695a40b"
Oct 30 13:30:23.773560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.773540    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"
Oct 30 13:30:23.773652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.773542    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd" event={"ID":"75bc6576-0613-4b84-a683-ed8afd915111","Type":"ContainerDied","Data":"3dde33671b9bbf8311fad65bd5c4ce8b74187dea0a45bc4c6aa5a5c35abfe44e"}
Oct 30 13:30:23.775110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.775091    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx" event={"ID":"6cb2c2fb-83d5-4af2-b030-5368808f2c82","Type":"ContainerDied","Data":"07af08c7adee25383a4382ccacbbf8b43ca0f96116d1f1e2b780e31111866727"}
Oct 30 13:30:23.775201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.775100    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"
Oct 30 13:30:23.777302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.777282    4993 generic.go:356] "Generic (PLEG): container finished" podID="187f5533-755c-4fda-b817-7e1d2b97defc" containerID="50bc6cacf53a920ece8cbaf6b2c520fe24f0bbcbee2786e99bbf9ff488468ab8" exitCode=60
Oct 30 13:30:23.777403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.777338    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" event={"ID":"187f5533-755c-4fda-b817-7e1d2b97defc","Type":"ContainerDied","Data":"50bc6cacf53a920ece8cbaf6b2c520fe24f0bbcbee2786e99bbf9ff488468ab8"}
Oct 30 13:30:23.777818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.777715    4993 scope.go:122] "RemoveContainer" containerID="50bc6cacf53a920ece8cbaf6b2c520fe24f0bbcbee2786e99bbf9ff488468ab8"
Oct 30 13:30:23.777986 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:23.777963    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4_odh-ai-gateway-infra(187f5533-755c-4fda-b817-7e1d2b97defc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" podUID="187f5533-755c-4fda-b817-7e1d2b97defc"
Oct 30 13:30:23.779154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.779133    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"
Oct 30 13:30:23.779620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.779599    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj" event={"ID":"77c140cb-a654-47db-b51e-a88e9455d374","Type":"ContainerDied","Data":"9de740bbe7226c124a7c31d1413d0d92a3d5b9d3e35ad3a9b9c4c005c083bf98"}
Oct 30 13:30:23.781342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.781321    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"
Oct 30 13:30:23.781457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.781321    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5" event={"ID":"465dbcb3-1211-4b3e-840c-19d89a3e119e","Type":"ContainerDied","Data":"ae9413812d1b3c15a7a70466973bbb22af79d071c856137526ff0abba6aee947"}
Oct 30 13:30:23.784019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.783980    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" event={"ID":"45be2541-5839-4419-afe3-3a747093066a","Type":"ContainerStarted","Data":"d96c0c837fe30f4a170a67c1955cec431ca502fa4eec4fab45eca3c251279e14"}
Oct 30 13:30:23.785112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.784640    4993 scope.go:122] "RemoveContainer" containerID="d96c0c837fe30f4a170a67c1955cec431ca502fa4eec4fab45eca3c251279e14"
Oct 30 13:30:23.785112 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:23.784892    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz_odh-ai-gateway-infra(45be2541-5839-4419-afe3-3a747093066a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" podUID="45be2541-5839-4419-afe3-3a747093066a"
Oct 30 13:30:23.786810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.786788    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"
Oct 30 13:30:23.786912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.786787    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w" event={"ID":"dd6514e0-bbbd-4933-836f-fc32631577ad","Type":"ContainerDied","Data":"2238808ec98c86e80d31a2ba9e39bf17150ac7ddf927031d900f1fbfb54f0541"}
Oct 30 13:30:23.791302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.791282    4993 scope.go:122] "RemoveContainer" containerID="10929dc888e829392456480bfa714cd46e2726c0f42665b578f38f7ce9bd84ed"
Oct 30 13:30:23.827322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.826978    4993 scope.go:122] "RemoveContainer" containerID="db08c9f657f5d577a0b9ae9662e1fbf5495f246754e3daf3bd02cc5caebd452a"
Oct 30 13:30:23.827322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.827044    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"]
Oct 30 13:30:23.844728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.844700    4993 scope.go:122] "RemoveContainer" containerID="19ca0e81a84e722c15e00637e02a914d77f11355e63092dd4034c5b05cca4e83"
Oct 30 13:30:23.913814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.913786    4993 scope.go:122] "RemoveContainer" containerID="bd5fe8ec4fca778de39e4067bc6dfeeab6fcc6b18c23b05eef47c34f86b88e64"
Oct 30 13:30:23.926496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.926474    4993 scope.go:122] "RemoveContainer" containerID="acb995bd54b9af5791b8d8e25689cd4b4344c5cfe44166c0437214ea1507ff93"
Oct 30 13:30:23.966923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.966773    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:23.967010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.966917    4993 scope.go:122] "RemoveContainer" containerID="f07b73a7dbb40c2225b45e37b1a3acac5384d949f7f8ddb96712ef58e4b4fa58"
Oct 30 13:30:23.977015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.976991    4993 scope.go:122] "RemoveContainer" containerID="f5fc3a7063c6da63ae650b39b2a31f220b5b39ec842c1cff29c3d14048d76362"
Oct 30 13:30:23.988240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:23.988175    4993 scope.go:122] "RemoveContainer" containerID="c56d933bbd36f203da51217e28016d65021850cbf9c32cdb471a347027d9e32c"
Oct 30 13:30:24.022964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.022929    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"]
Oct 30 13:30:24.028343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.028310    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889450-gthr4"]
Oct 30 13:30:24.061520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.061486    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"]
Oct 30 13:30:24.066411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.066380    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889450-c89q5"]
Oct 30 13:30:24.066877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.066850    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14cbc6e-1967-47a3-be75-b2251ecc25ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14cbc6e-1967-47a3-be75-b2251ecc25ce-maas-api-service-ca\") pod \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\" (UID: \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\") "
Oct 30 13:30:24.067026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.067013    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14cbc6e-1967-47a3-be75-b2251ecc25ce-kube-api-access-nmzgd\" (UniqueName: \"kubernetes.io/projected/d14cbc6e-1967-47a3-be75-b2251ecc25ce-kube-api-access-nmzgd\") pod \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\" (UID: \"d14cbc6e-1967-47a3-be75-b2251ecc25ce\") "
Oct 30 13:30:24.067557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.067533    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14cbc6e-1967-47a3-be75-b2251ecc25ce-maas-api-service-ca" pod "d14cbc6e-1967-47a3-be75-b2251ecc25ce" (UID: "d14cbc6e-1967-47a3-be75-b2251ecc25ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:24.069385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.069362    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14cbc6e-1967-47a3-be75-b2251ecc25ce-kube-api-access-nmzgd" pod "d14cbc6e-1967-47a3-be75-b2251ecc25ce" (UID: "d14cbc6e-1967-47a3-be75-b2251ecc25ce"). InnerVolumeSpecName "kube-api-access-nmzgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:24.098256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.098176    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"]
Oct 30 13:30:24.103268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.103234    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889450-g2r7w"]
Oct 30 13:30:24.136676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.136641    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"]
Oct 30 13:30:24.141708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.141677    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889450-mv7cj"]
Oct 30 13:30:24.168330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.168293    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14cbc6e-1967-47a3-be75-b2251ecc25ce-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:24.168330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.168323    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzgd\" (UniqueName: \"kubernetes.io/projected/d14cbc6e-1967-47a3-be75-b2251ecc25ce-kube-api-access-nmzgd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:24.200667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.200622    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"]
Oct 30 13:30:24.215545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.215485    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"]
Oct 30 13:30:24.235165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.235116    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889450-tw8kd"]
Oct 30 13:30:24.293620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.293575    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"]
Oct 30 13:30:24.301129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.301095    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"]
Oct 30 13:30:24.307618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.307583    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889450-jbsbx"]
Oct 30 13:30:24.806022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.805990    4993 generic.go:356] "Generic (PLEG): container finished" podID="45be2541-5839-4419-afe3-3a747093066a" containerID="d96c0c837fe30f4a170a67c1955cec431ca502fa4eec4fab45eca3c251279e14" exitCode=60
Oct 30 13:30:24.806496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.806058    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" event={"ID":"45be2541-5839-4419-afe3-3a747093066a","Type":"ContainerDied","Data":"d96c0c837fe30f4a170a67c1955cec431ca502fa4eec4fab45eca3c251279e14"}
Oct 30 13:30:24.806496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.806111    4993 scope.go:122] "RemoveContainer" containerID="9e7d915ff4f45a766450130afb0bad219c60973e9be2cbcf342e7954772dcc2c"
Oct 30 13:30:24.806986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.806968    4993 scope.go:122] "RemoveContainer" containerID="d96c0c837fe30f4a170a67c1955cec431ca502fa4eec4fab45eca3c251279e14"
Oct 30 13:30:24.807396 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:24.807373    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz_odh-ai-gateway-infra(45be2541-5839-4419-afe3-3a747093066a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" podUID="45be2541-5839-4419-afe3-3a747093066a"
Oct 30 13:30:24.814444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.814419    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5" event={"ID":"d14cbc6e-1967-47a3-be75-b2251ecc25ce","Type":"ContainerDied","Data":"22cb5620364fc6044163247d95c9b9ea8121684372823a0c11f416692e428dfc"}
Oct 30 13:30:24.814542 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.814528    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"
Oct 30 13:30:24.825756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.825733    4993 scope.go:122] "RemoveContainer" containerID="03f58b1c6af38a440a1482e7d3d307c09bdbe01ea1a7ac6704a45022dd557c65"
Oct 30 13:30:24.896322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.888323    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"]
Oct 30 13:30:24.916313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:24.916271    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889450-ph9t5"]
Oct 30 13:30:25.030597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.030345    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:25.079160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.079093    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:25.079881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.079856    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187f5533-755c-4fda-b817-7e1d2b97defc-kube-api-access-kfktq\" (UniqueName: \"kubernetes.io/projected/187f5533-755c-4fda-b817-7e1d2b97defc-kube-api-access-kfktq\") pod \"187f5533-755c-4fda-b817-7e1d2b97defc\" (UID: \"187f5533-755c-4fda-b817-7e1d2b97defc\") "
Oct 30 13:30:25.079987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.079968    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187f5533-755c-4fda-b817-7e1d2b97defc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187f5533-755c-4fda-b817-7e1d2b97defc-maas-api-service-ca\") pod \"187f5533-755c-4fda-b817-7e1d2b97defc\" (UID: \"187f5533-755c-4fda-b817-7e1d2b97defc\") "
Oct 30 13:30:25.081062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.081036    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187f5533-755c-4fda-b817-7e1d2b97defc-maas-api-service-ca" pod "187f5533-755c-4fda-b817-7e1d2b97defc" (UID: "187f5533-755c-4fda-b817-7e1d2b97defc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:25.090527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.090389    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187f5533-755c-4fda-b817-7e1d2b97defc-kube-api-access-kfktq" pod "187f5533-755c-4fda-b817-7e1d2b97defc" (UID: "187f5533-755c-4fda-b817-7e1d2b97defc"). InnerVolumeSpecName "kube-api-access-kfktq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:25.111572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.111542    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:25.182232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.181393    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-kube-api-access-dvz5d\" (UniqueName: \"kubernetes.io/projected/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-kube-api-access-dvz5d\") pod \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\" (UID: \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\") "
Oct 30 13:30:25.182232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.181521    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86002868-e959-4e93-9a73-a6ca9695a40b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86002868-e959-4e93-9a73-a6ca9695a40b-maas-api-service-ca\") pod \"86002868-e959-4e93-9a73-a6ca9695a40b\" (UID: \"86002868-e959-4e93-9a73-a6ca9695a40b\") "
Oct 30 13:30:25.182232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.181571    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-maas-api-service-ca\") pod \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\" (UID: \"ce27ddf5-ef65-427e-8793-bfbc3015ddbe\") "
Oct 30 13:30:25.182232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.181787    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86002868-e959-4e93-9a73-a6ca9695a40b-kube-api-access-mrr6m\" (UniqueName: \"kubernetes.io/projected/86002868-e959-4e93-9a73-a6ca9695a40b-kube-api-access-mrr6m\") pod \"86002868-e959-4e93-9a73-a6ca9695a40b\" (UID: \"86002868-e959-4e93-9a73-a6ca9695a40b\") "
Oct 30 13:30:25.182232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.182029    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86002868-e959-4e93-9a73-a6ca9695a40b-maas-api-service-ca" pod "86002868-e959-4e93-9a73-a6ca9695a40b" (UID: "86002868-e959-4e93-9a73-a6ca9695a40b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:25.182564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.182246    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-maas-api-service-ca" pod "ce27ddf5-ef65-427e-8793-bfbc3015ddbe" (UID: "ce27ddf5-ef65-427e-8793-bfbc3015ddbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:25.182564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.182465    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187f5533-755c-4fda-b817-7e1d2b97defc-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:25.182564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.182487    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86002868-e959-4e93-9a73-a6ca9695a40b-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:25.182564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.182501    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:25.182564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.182516    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfktq\" (UniqueName: \"kubernetes.io/projected/187f5533-755c-4fda-b817-7e1d2b97defc-kube-api-access-kfktq\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:25.187250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.184754    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86002868-e959-4e93-9a73-a6ca9695a40b-kube-api-access-mrr6m" pod "86002868-e959-4e93-9a73-a6ca9695a40b" (UID: "86002868-e959-4e93-9a73-a6ca9695a40b"). InnerVolumeSpecName "kube-api-access-mrr6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:25.187250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.184821    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-kube-api-access-dvz5d" pod "ce27ddf5-ef65-427e-8793-bfbc3015ddbe" (UID: "ce27ddf5-ef65-427e-8793-bfbc3015ddbe"). InnerVolumeSpecName "kube-api-access-dvz5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:25.264480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.264439    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"]
Oct 30 13:30:25.284410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.284372    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrr6m\" (UniqueName: \"kubernetes.io/projected/86002868-e959-4e93-9a73-a6ca9695a40b-kube-api-access-mrr6m\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:25.284410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.284401    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvz5d\" (UniqueName: \"kubernetes.io/projected/ce27ddf5-ef65-427e-8793-bfbc3015ddbe-kube-api-access-dvz5d\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:25.695696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.695658    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" path="/var/lib/kubelet/pods/465dbcb3-1211-4b3e-840c-19d89a3e119e/volumes"
Oct 30 13:30:25.696302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.696283    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" path="/var/lib/kubelet/pods/6cb2c2fb-83d5-4af2-b030-5368808f2c82/volumes"
Oct 30 13:30:25.696846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.696829    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75bc6576-0613-4b84-a683-ed8afd915111" path="/var/lib/kubelet/pods/75bc6576-0613-4b84-a683-ed8afd915111/volumes"
Oct 30 13:30:25.697808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.697789    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c140cb-a654-47db-b51e-a88e9455d374" path="/var/lib/kubelet/pods/77c140cb-a654-47db-b51e-a88e9455d374/volumes"
Oct 30 13:30:25.698370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.698354    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" path="/var/lib/kubelet/pods/bdcaa358-f46d-4b45-97bb-3e617acd4ed8/volumes"
Oct 30 13:30:25.698903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.698880    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" path="/var/lib/kubelet/pods/d14cbc6e-1967-47a3-be75-b2251ecc25ce/volumes"
Oct 30 13:30:25.699820 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.699782    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" path="/var/lib/kubelet/pods/dd6514e0-bbbd-4933-836f-fc32631577ad/volumes"
Oct 30 13:30:25.839470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.839438    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"
Oct 30 13:30:25.839976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.839448    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9" event={"ID":"ce27ddf5-ef65-427e-8793-bfbc3015ddbe","Type":"ContainerDied","Data":"7756cdd946cdde69a3f0d599192ce9acc33549b39a3141ce94fdb3c21be96223"}
Oct 30 13:30:25.839976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.839574    4993 scope.go:122] "RemoveContainer" containerID="de792354897b2322256e39beb6115f50da3fe1c10ed0ead90dc9f7cd7ac8c1b3"
Oct 30 13:30:25.842442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.842404    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"
Oct 30 13:30:25.842568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.842447    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp" event={"ID":"86002868-e959-4e93-9a73-a6ca9695a40b","Type":"ContainerDied","Data":"5361c8d16955485385674e628ec5b1fac91627ffb335862ea352d06f8933f076"}
Oct 30 13:30:25.844120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.844100    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"
Oct 30 13:30:25.844120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.844105    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4" event={"ID":"187f5533-755c-4fda-b817-7e1d2b97defc","Type":"ContainerDied","Data":"86c8f5b3c56bc6cce1801635eecb0dd200b296d3fb885cfc5ae845bfa4c5a3f3"}
Oct 30 13:30:25.851996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.851969    4993 scope.go:122] "RemoveContainer" containerID="4fa38d59de6c621397a608e8c50d14122fa63094003d558f27bea02bfb66b6f1"
Oct 30 13:30:25.863331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.863302    4993 scope.go:122] "RemoveContainer" containerID="50bc6cacf53a920ece8cbaf6b2c520fe24f0bbcbee2786e99bbf9ff488468ab8"
Oct 30 13:30:25.946637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.946550    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"]
Oct 30 13:30:25.966253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.957002    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889450-59rqp"]
Oct 30 13:30:25.999689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.992172    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"]
Oct 30 13:30:25.999689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:25.997690    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889450-227t4"]
Oct 30 13:30:26.037986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.037958    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:26.042295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.042268    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"]
Oct 30 13:30:26.046587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.046560    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889450-dk4j9"]
Oct 30 13:30:26.096489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.096457    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45be2541-5839-4419-afe3-3a747093066a-kube-api-access-2bclb\" (UniqueName: \"kubernetes.io/projected/45be2541-5839-4419-afe3-3a747093066a-kube-api-access-2bclb\") pod \"45be2541-5839-4419-afe3-3a747093066a\" (UID: \"45be2541-5839-4419-afe3-3a747093066a\") "
Oct 30 13:30:26.096882 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.096851    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45be2541-5839-4419-afe3-3a747093066a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45be2541-5839-4419-afe3-3a747093066a-maas-api-service-ca\") pod \"45be2541-5839-4419-afe3-3a747093066a\" (UID: \"45be2541-5839-4419-afe3-3a747093066a\") "
Oct 30 13:30:26.097470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.097444    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45be2541-5839-4419-afe3-3a747093066a-maas-api-service-ca" pod "45be2541-5839-4419-afe3-3a747093066a" (UID: "45be2541-5839-4419-afe3-3a747093066a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:26.101617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.101592    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45be2541-5839-4419-afe3-3a747093066a-kube-api-access-2bclb" pod "45be2541-5839-4419-afe3-3a747093066a" (UID: "45be2541-5839-4419-afe3-3a747093066a"). InnerVolumeSpecName "kube-api-access-2bclb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:26.199003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.198907    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bclb\" (UniqueName: \"kubernetes.io/projected/45be2541-5839-4419-afe3-3a747093066a-kube-api-access-2bclb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:26.199003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.198941    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45be2541-5839-4419-afe3-3a747093066a-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:26.861697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.861661    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"
Oct 30 13:30:26.861697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.861663    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz" event={"ID":"45be2541-5839-4419-afe3-3a747093066a","Type":"ContainerDied","Data":"817188b66efde3d04eeb5e52fa870b279a86622c4432c29580104ba063dd3a36"}
Oct 30 13:30:26.862293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.861732    4993 scope.go:122] "RemoveContainer" containerID="d96c0c837fe30f4a170a67c1955cec431ca502fa4eec4fab45eca3c251279e14"
Oct 30 13:30:26.936487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.936451    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"]
Oct 30 13:30:26.948095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:26.948060    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889450-fxqpz"]
Oct 30 13:30:27.700782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:27.700743    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" path="/var/lib/kubelet/pods/187f5533-755c-4fda-b817-7e1d2b97defc/volumes"
Oct 30 13:30:27.701404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:27.701386    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45be2541-5839-4419-afe3-3a747093066a" path="/var/lib/kubelet/pods/45be2541-5839-4419-afe3-3a747093066a/volumes"
Oct 30 13:30:27.701792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:27.701779    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" path="/var/lib/kubelet/pods/86002868-e959-4e93-9a73-a6ca9695a40b/volumes"
Oct 30 13:30:27.702510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:27.702495    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" path="/var/lib/kubelet/pods/ce27ddf5-ef65-427e-8793-bfbc3015ddbe/volumes"
Oct 30 13:30:28.046376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.046334    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"]
Oct 30 13:30:28.046957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.046938    4993 cpu_manager.go:414] "containerMap: removing container" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.046957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.046956    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.046968    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.046973    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.046983    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.046988    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.046995    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047000    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047008    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047014    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047023    4993 cpu_manager.go:414] "containerMap: removing container" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047028    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047033    4993 cpu_manager.go:414] "containerMap: removing container" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047037    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047045    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047050    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047058    4993 cpu_manager.go:414] "containerMap: removing container" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047062    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047070    4993 cpu_manager.go:414] "containerMap: removing container" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047075    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047082    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047087    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047094    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047099    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047106    4993 cpu_manager.go:414] "containerMap: removing container" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047110    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047117    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047121    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047126    4993 cpu_manager.go:414] "containerMap: removing container" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:28.047112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047131    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047140    4993 cpu_manager.go:414] "containerMap: removing container" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047145    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047153    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047158    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047163    4993 cpu_manager.go:414] "containerMap: removing container" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047168    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047175    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047179    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047185    4993 cpu_manager.go:414] "containerMap: removing container" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047190    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047199    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047217    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047226    4993 cpu_manager.go:414] "containerMap: removing container" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047230    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047237    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047241    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047247    4993 cpu_manager.go:414] "containerMap: removing container" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047251    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047257    4993 cpu_manager.go:414] "containerMap: removing container" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047261    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047270    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047274    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047281    4993 cpu_manager.go:414] "containerMap: removing container" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047285    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047291    4993 cpu_manager.go:414] "containerMap: removing container" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047295    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047307    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047312    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047321    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047326    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047333    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047337    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047344    4993 cpu_manager.go:414] "containerMap: removing container" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047349    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047356    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047360    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047367    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047372    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047379    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047383    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047392    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047397    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047406    4993 cpu_manager.go:414] "containerMap: removing container" podUID="986a714d-458c-47dc-b878-79c74ffa9979" containerName="revoke-keys"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047418    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986a714d-458c-47dc-b878-79c74ffa9979" containerName="revoke-keys"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047425    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047429    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047438    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1532e5a1-3417-487b-95b4-947fb71b20a4" containerName="payload-processing"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047442    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1532e5a1-3417-487b-95b4-947fb71b20a4" containerName="payload-processing"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047450    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047454    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047463    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047467    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047474    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047478    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047487    4993 cpu_manager.go:414] "containerMap: removing container" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047492    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047503    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047507    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047599    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047607    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047614    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047619    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047625    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047629    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047635    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047643    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047649    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047654    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047659    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047664    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047669    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047673    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047678    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047683    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047689    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047693    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047697    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047703    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047708    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047713    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047718    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047724    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caff419-28cc-4098-a183-45af8e982db0" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047730    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047735    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="465dbcb3-1211-4b3e-840c-19d89a3e119e" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047741    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047746    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14cbc6e-1967-47a3-be75-b2251ecc25ce" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047753    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047759    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047764    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="986a714d-458c-47dc-b878-79c74ffa9979" containerName="revoke-keys"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047769    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047776    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1532e5a1-3417-487b-95b4-947fb71b20a4" containerName="payload-processing"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047782    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18ca0c2-40ad-4793-982f-176d4efbf373" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047787    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047792    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047797    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047803    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047808    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047814    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="86002868-e959-4e93-9a73-a6ca9695a40b" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047819    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047824    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047829    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047834    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:28.048587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.047840    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:28.053329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.053304    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.086843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.086781    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"]
Oct 30 13:30:28.123625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.123579    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks84l\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-kube-api-access-ks84l\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.124014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.123992    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.124153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.124118    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.124321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.124295    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.124403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.124345    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.124403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.124392    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.124486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.124416    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.124486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.124445    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.124572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.124504    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb718195-5572-418f-a6be-e3fa3a0fccad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.225930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.225889    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.225996    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226370    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226380    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226446    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226508 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226482    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226506    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226537    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226598    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb718195-5572-418f-a6be-e3fa3a0fccad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226610    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.226893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.226739    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks84l\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-kube-api-access-ks84l\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.227689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.227660    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb718195-5572-418f-a6be-e3fa3a0fccad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.228079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.228053    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.228201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.228176    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.229710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.229686    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.229984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.229965    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.253335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.253299    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks84l\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-kube-api-access-ks84l\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.254659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.254628    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-28wtn\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.391286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.383487    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:28.776295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.776264    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"]
Oct 30 13:30:28.779784 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:30:28.779743    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb718195_5572_418f_a6be_e3fa3a0fccad.slice/crio-de3b3242bc1526615979304b17322a117e93175909c7952eba4f5bf0f99e675a WatchSource:0}: Error finding container de3b3242bc1526615979304b17322a117e93175909c7952eba4f5bf0f99e675a: Status 404 returned error can't find the container with id de3b3242bc1526615979304b17322a117e93175909c7952eba4f5bf0f99e675a
Oct 30 13:30:28.783391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.783331    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:28.783505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.783457    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:28.783543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.783503    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:28.922696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.916638    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" event={"ID":"bb718195-5572-418f-a6be-e3fa3a0fccad","Type":"ContainerStarted","Data":"04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06"}
Oct 30 13:30:28.922696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:28.916774    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" event={"ID":"bb718195-5572-418f-a6be-e3fa3a0fccad","Type":"ContainerStarted","Data":"de3b3242bc1526615979304b17322a117e93175909c7952eba4f5bf0f99e675a"}
Oct 30 13:30:29.026560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:29.015542    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podStartSLOduration=1.015518671 podStartE2EDuration="1.015518671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:28.974681403 +0000 UTC m=+3493.450401017" watchObservedRunningTime="2026-10-30 13:30:29.015518671 +0000 UTC m=+3493.491238282"
Oct 30 13:30:29.392764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:29.386600    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:29.403189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:29.397386    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:29.403189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:29.397462    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:30.395232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.385120    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:30.395232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.385185    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:30.539117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539073    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"]
Oct 30 13:30:30.539916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539890    4993 cpu_manager.go:414] "containerMap: removing container" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539918    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539941    4993 cpu_manager.go:414] "containerMap: removing container" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539947    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539959    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539965    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539983    4993 cpu_manager.go:414] "containerMap: removing container" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.539992    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540004    4993 cpu_manager.go:414] "containerMap: removing container" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540009    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a20603-e044-4279-a9e8-7c2c7d030a44" containerName="cleanup"
Oct 30 13:30:30.540018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540021    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540026    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce27ddf5-ef65-427e-8793-bfbc3015ddbe" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540045    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540049    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af816ef-1abb-4490-a7e0-4d1901c27223" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540058    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540063    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540080    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540086    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ec16c9-b636-4241-bca4-6bcaf05e9f11" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540180    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb2c2fb-83d5-4af2-b030-5368808f2c82" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540189    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c140cb-a654-47db-b51e-a88e9455d374" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540199    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6514e0-bbbd-4933-836f-fc32631577ad" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540249    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcaa358-f46d-4b45-97bb-3e617acd4ed8" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540271    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="45be2541-5839-4419-afe3-3a747093066a" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540278    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0fb7ae-75e3-43e9-bedf-69e2a76e7da3" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540284    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bc6576-0613-4b84-a683-ed8afd915111" containerName="cleanup"
Oct 30 13:30:30.540419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.540297    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="187f5533-755c-4fda-b817-7e1d2b97defc" containerName="cleanup"
Oct 30 13:30:30.543917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.543897    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.571498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.571463    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"kube-root-ca.crt\""
Oct 30 13:30:30.571745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.571584    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-88e9f3\""
Oct 30 13:30:30.582246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.571896    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"payload-processing-external-model-e2e--69d1522b-dockercfg-fjjbd\""
Oct 30 13:30:30.582246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.571899    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"agc-cred-c30a94579b6b\""
Oct 30 13:30:30.582246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.571964    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"routing-overlay\""
Oct 30 13:30:30.582246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.572027    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"openshift-service-ca.crt\""
Oct 30 13:30:30.582246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.579378    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"]
Oct 30 13:30:30.610846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.610793    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-kube-api-access-p92v6\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.611023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.610986    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.611187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.611042    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.611269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.611250    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.726250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.726145    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-kube-api-access-p92v6\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.736489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.736443    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.736767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.736745    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.737087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.737060    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.766229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.738521    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.766480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.740175    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.788234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.780589    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-kube-api-access-p92v6\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.789739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.789708    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:30.866992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:30.865635    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:31.320245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:31.311843    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"]
Oct 30 13:30:31.321771 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:30:31.321731    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3647e0ae_cf37_400e_b943_ceed09f879cd.slice/crio-011fe78433ddf3687bbfe62426eb5554363869d938f141469648882035defcb4 WatchSource:0}: Error finding container 011fe78433ddf3687bbfe62426eb5554363869d938f141469648882035defcb4: Status 404 returned error can't find the container with id 011fe78433ddf3687bbfe62426eb5554363869d938f141469648882035defcb4
Oct 30 13:30:31.384524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:31.383849    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:31.384524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:31.383905    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:32.006534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:32.006485    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n" event={"ID":"3647e0ae-cf37-400e-b943-ceed09f879cd","Type":"ContainerStarted","Data":"b4c58072e0691b9545a5a9fd8e7d3ff4a8b636558e1caeb49c7383af5c9b8728"}
Oct 30 13:30:32.006969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:32.006549    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n" event={"ID":"3647e0ae-cf37-400e-b943-ceed09f879cd","Type":"ContainerStarted","Data":"011fe78433ddf3687bbfe62426eb5554363869d938f141469648882035defcb4"}
Oct 30 13:30:32.006969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:32.006626    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:32.049168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:32.049099    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n" podStartSLOduration=2.049080484 podStartE2EDuration="2.049080484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:32.047449655 +0000 UTC m=+3496.523169265" watchObservedRunningTime="2026-10-30 13:30:32.049080484 +0000 UTC m=+3496.524800095"
Oct 30 13:30:32.384277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:32.384136    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:32.384277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:32.384259    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:33.385682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:33.385642    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:33.386284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:33.385702    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:34.383909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:34.383869    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:34.384101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:34.384019    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:35.384817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:35.384780    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:35.385343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:35.384842    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:36.385246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:36.384817    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:36.385246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:36.384889    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:37.384654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:37.384612    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:37.385107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:37.385067    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:38.386234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:38.385679    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:38.386234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:38.385752    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:38.825853 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:38.825810    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:30:39.385233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:39.384736    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-28wtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:30:39.385233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:39.384807    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused"
Oct 30 13:30:39.525456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:39.525408    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgkz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:30:39.526006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:39.525482    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgkz8" podUID="d33694bd-a804-4182-8d4b-e30156066d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:40.780371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:40.780327    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-k879q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:30:40.781057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:40.781018    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:41.027537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.027485    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"]
Oct 30 13:30:41.033468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.033384    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.053228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.053167    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"]
Oct 30 13:30:41.057577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.057538    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:30:41.058289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.057885    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:30:41.058456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.057932    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:30:41.058562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.058060    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:30:41.147752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.147677    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxn25\" (UniqueName: \"kubernetes.io/projected/d7367a9b-35e7-4115-93d8-de704cc5a5c6-kube-api-access-fxn25\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.147983 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.147824    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.148094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.148073    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.242409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.242377    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:41.243080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.243053    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:41.244013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.243989    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:30:41.249810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.249776    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxn25\" (UniqueName: \"kubernetes.io/projected/d7367a9b-35e7-4115-93d8-de704cc5a5c6-kube-api-access-fxn25\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.249963 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.249864    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.250137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.250122    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.250803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.250780    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.252668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.252642    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.274761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.274722    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxn25\" (UniqueName: \"kubernetes.io/projected/d7367a9b-35e7-4115-93d8-de704cc5a5c6-kube-api-access-fxn25\") pod \"maas-api-revoke-sub-a994c677-mbccl\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.350978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.350892    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:41.704247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:41.701581    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"]
Oct 30 13:30:41.710251 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:30:41.710184    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7367a9b_35e7_4115_93d8_de704cc5a5c6.slice/crio-9ffec28b3e7ec939777c7c4da56b7c7813340c30dd66e318218a8c26147fe45b WatchSource:0}: Error finding container 9ffec28b3e7ec939777c7c4da56b7c7813340c30dd66e318218a8c26147fe45b: Status 404 returned error can't find the container with id 9ffec28b3e7ec939777c7c4da56b7c7813340c30dd66e318218a8c26147fe45b
Oct 30 13:30:42.263760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:42.263646    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl" event={"ID":"d7367a9b-35e7-4115-93d8-de704cc5a5c6","Type":"ContainerStarted","Data":"2d54816471e133b138e6fd299783fa1497a946f83d167f496341a4159ee53617"}
Oct 30 13:30:42.264271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:42.263763    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl" event={"ID":"d7367a9b-35e7-4115-93d8-de704cc5a5c6","Type":"ContainerStarted","Data":"9ffec28b3e7ec939777c7c4da56b7c7813340c30dd66e318218a8c26147fe45b"}
Oct 30 13:30:42.305657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:42.305572    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl" podStartSLOduration=2.305551795 podStartE2EDuration="2.305551795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:42.303974134 +0000 UTC m=+3506.779693744" watchObservedRunningTime="2026-10-30 13:30:42.305551795 +0000 UTC m=+3506.781271407"
Oct 30 13:30:43.020325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:43.020289    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:30:43.284125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:43.284088    4993 generic.go:356] "Generic (PLEG): container finished" podID="d7367a9b-35e7-4115-93d8-de704cc5a5c6" containerID="2d54816471e133b138e6fd299783fa1497a946f83d167f496341a4159ee53617" exitCode=0
Oct 30 13:30:43.284692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:43.284326    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl" event={"ID":"d7367a9b-35e7-4115-93d8-de704cc5a5c6","Type":"ContainerDied","Data":"2d54816471e133b138e6fd299783fa1497a946f83d167f496341a4159ee53617"}
Oct 30 13:30:44.487776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.487739    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:44.611498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.611383    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7367a9b-35e7-4115-93d8-de704cc5a5c6-kube-api-access-fxn25\" (UniqueName: \"kubernetes.io/projected/d7367a9b-35e7-4115-93d8-de704cc5a5c6-kube-api-access-fxn25\") pod \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") "
Oct 30 13:30:44.611692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.611618    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-openshift-ca\") pod \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") "
Oct 30 13:30:44.611795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.611779    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-xks-ca\") pod \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\" (UID: \"d7367a9b-35e7-4115-93d8-de704cc5a5c6\") "
Oct 30 13:30:44.612024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.611996    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-openshift-ca" pod "d7367a9b-35e7-4115-93d8-de704cc5a5c6" (UID: "d7367a9b-35e7-4115-93d8-de704cc5a5c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:44.612376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.612357    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:44.613703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.613673    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7367a9b-35e7-4115-93d8-de704cc5a5c6-kube-api-access-fxn25" pod "d7367a9b-35e7-4115-93d8-de704cc5a5c6" (UID: "d7367a9b-35e7-4115-93d8-de704cc5a5c6"). InnerVolumeSpecName "kube-api-access-fxn25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:44.613814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.613759    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-xks-ca" pod "d7367a9b-35e7-4115-93d8-de704cc5a5c6" (UID: "d7367a9b-35e7-4115-93d8-de704cc5a5c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:30:44.714322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.714260    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7367a9b-35e7-4115-93d8-de704cc5a5c6-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:44.714322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:44.714312    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxn25\" (UniqueName: \"kubernetes.io/projected/d7367a9b-35e7-4115-93d8-de704cc5a5c6-kube-api-access-fxn25\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:45.302375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:45.302341    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"
Oct 30 13:30:45.302584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:45.302378    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl" event={"ID":"d7367a9b-35e7-4115-93d8-de704cc5a5c6","Type":"ContainerDied","Data":"9ffec28b3e7ec939777c7c4da56b7c7813340c30dd66e318218a8c26147fe45b"}
Oct 30 13:30:45.302584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:45.302436    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ffec28b3e7ec939777c7c4da56b7c7813340c30dd66e318218a8c26147fe45b"
Oct 30 13:30:53.949159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.949119    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"]
Oct 30 13:30:53.949861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.949842    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d7367a9b-35e7-4115-93d8-de704cc5a5c6" containerName="revoke-keys"
Oct 30 13:30:53.949861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.949862    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7367a9b-35e7-4115-93d8-de704cc5a5c6" containerName="revoke-keys"
Oct 30 13:30:53.950037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.950027    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7367a9b-35e7-4115-93d8-de704cc5a5c6" containerName="revoke-keys"
Oct 30 13:30:53.953804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.953780    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:53.970973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.970936    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"agc-cred-fed1fdea6b03\""
Oct 30 13:30:53.982803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.982747    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-e2e--8217e17a-dockercfg-hd5xv\""
Oct 30 13:30:53.983002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.982970    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"openshift-service-ca.crt\""
Oct 30 13:30:53.983708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.983675    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"kube-root-ca.crt\""
Oct 30 13:30:53.983846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.983747    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1957dc\""
Oct 30 13:30:53.983846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.983808    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"routing-overlay\""
Oct 30 13:30:53.992968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:53.992932    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"]
Oct 30 13:30:54.016331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.016288    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.016555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.016456    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.016665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.016631    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.016736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.016714    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wtvr\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-kube-api-access-6wtvr\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.124249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.119257    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wtvr\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-kube-api-access-6wtvr\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.124249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.119400    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.124249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.119549    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.124249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.119689    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.124249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.120598    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.124249 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:54.120991    4993 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-fed1fdea6b03: secret "agc-cred-fed1fdea6b03" not found
Oct 30 13:30:54.124249 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:54.121010    4993 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w: secret "agc-cred-fed1fdea6b03" not found
Oct 30 13:30:54.124249 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:54.121100    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-provider-credentials podName:04bf9f4e-2cee-48f4-bed4-998cc3a43429 nodeName:}" failed. No retries permitted until 2026-10-30 13:30:54.621079349 +0000 UTC m=+3519.096798956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w" (UID: "04bf9f4e-2cee-48f4-bed4-998cc3a43429") : secret "agc-cred-fed1fdea6b03" not found
Oct 30 13:30:54.140706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.140658    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.215071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.214989    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wtvr\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-kube-api-access-6wtvr\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.231979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.231936    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"]
Oct 30 13:30:54.232757 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:54.232723    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w" podUID="04bf9f4e-2cee-48f4-bed4-998cc3a43429"
Oct 30 13:30:54.394382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.394343    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.403765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.403736    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.424340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.424309    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-routing-overlay-volume\") pod \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") "
Oct 30 13:30:54.424528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.424421    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-kube-api-access-6wtvr\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-kube-api-access-6wtvr\") pod \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") "
Oct 30 13:30:54.424573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.424543    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-plugins-config-volume\") pod \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") "
Oct 30 13:30:54.424775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.424747    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-routing-overlay-volume" pod "04bf9f4e-2cee-48f4-bed4-998cc3a43429" (UID: "04bf9f4e-2cee-48f4-bed4-998cc3a43429"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:54.424974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.424943    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-plugins-config-volume" pod "04bf9f4e-2cee-48f4-bed4-998cc3a43429" (UID: "04bf9f4e-2cee-48f4-bed4-998cc3a43429"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:54.425110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.425045    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:54.426587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.426569    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-kube-api-access-6wtvr" pod "04bf9f4e-2cee-48f4-bed4-998cc3a43429" (UID: "04bf9f4e-2cee-48f4-bed4-998cc3a43429"). InnerVolumeSpecName "kube-api-access-6wtvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:54.539193 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.539147    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bf9f4e-2cee-48f4-bed4-998cc3a43429-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:54.554474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.554429    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wtvr\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-kube-api-access-6wtvr\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:54.662231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:54.659159    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w\" (UID: \"04bf9f4e-2cee-48f4-bed4-998cc3a43429\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:54.662231 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:54.659684    4993 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-fed1fdea6b03: secret "agc-cred-fed1fdea6b03" not found
Oct 30 13:30:54.662231 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:54.659704    4993 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w: secret "agc-cred-fed1fdea6b03" not found
Oct 30 13:30:54.662231 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:30:54.659767    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-provider-credentials podName:04bf9f4e-2cee-48f4-bed4-998cc3a43429 nodeName:}" failed. No retries permitted until 2026-10-30 13:30:55.659748586 +0000 UTC m=+3520.135468180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w" (UID: "04bf9f4e-2cee-48f4-bed4-998cc3a43429") : secret "agc-cred-fed1fdea6b03" not found
Oct 30 13:30:55.404982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:55.404938    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"
Oct 30 13:30:55.495546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:55.495496    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"]
Oct 30 13:30:55.502319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:55.502270    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd558f4w"]
Oct 30 13:30:55.574748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:55.574705    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04bf9f4e-2cee-48f4-bed4-998cc3a43429-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:30:55.717555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:30:55.717467    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04bf9f4e-2cee-48f4-bed4-998cc3a43429" path="/var/lib/kubelet/pods/04bf9f4e-2cee-48f4-bed4-998cc3a43429/volumes"
Oct 30 13:31:03.305105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.305044    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"]
Oct 30 13:31:03.311074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.311038    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.331342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.331305    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"]
Oct 30 13:31:03.336472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.336439    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:31:03.336472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.336466    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:31:03.337045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.336913    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:31:03.337045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.337019    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:31:03.439729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.439682    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.439931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.439824    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xqp\" (UniqueName: \"kubernetes.io/projected/8a62756c-45ea-4b59-9c74-5781f0ff4afc-kube-api-access-z6xqp\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.440030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.440014    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.541737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.541698    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.541964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.541833    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.541964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.541911    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xqp\" (UniqueName: \"kubernetes.io/projected/8a62756c-45ea-4b59-9c74-5781f0ff4afc-kube-api-access-z6xqp\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.542461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.542437    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.544299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.544278    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.566154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.566076    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xqp\" (UniqueName: \"kubernetes.io/projected/8a62756c-45ea-4b59-9c74-5781f0ff4afc-kube-api-access-z6xqp\") pod \"maas-api-revoke-sub-9503e118-djjq6\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.626592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.626555    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:03.941962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:03.941932    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"]
Oct 30 13:31:03.944076 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:31:03.944042    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a62756c_45ea_4b59_9c74_5781f0ff4afc.slice/crio-709e7d4a2828d4bc4433718fadd590accc593cd7e56275b3d56f89e958981d69 WatchSource:0}: Error finding container 709e7d4a2828d4bc4433718fadd590accc593cd7e56275b3d56f89e958981d69: Status 404 returned error can't find the container with id 709e7d4a2828d4bc4433718fadd590accc593cd7e56275b3d56f89e958981d69
Oct 30 13:31:04.489015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:04.488970    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6" event={"ID":"8a62756c-45ea-4b59-9c74-5781f0ff4afc","Type":"ContainerStarted","Data":"7ed555867e236db6303c37400c97a3a4d77a5e784d7405773c82483dda71efe7"}
Oct 30 13:31:04.489568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:04.489041    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6" event={"ID":"8a62756c-45ea-4b59-9c74-5781f0ff4afc","Type":"ContainerStarted","Data":"709e7d4a2828d4bc4433718fadd590accc593cd7e56275b3d56f89e958981d69"}
Oct 30 13:31:04.531816 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:04.531683    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6" podStartSLOduration=1.531668027 podStartE2EDuration="1.531668027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:04.528899938 +0000 UTC m=+3529.004619549" watchObservedRunningTime="2026-10-30 13:31:04.531668027 +0000 UTC m=+3529.007387638"
Oct 30 13:31:05.502519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:05.502481    4993 generic.go:356] "Generic (PLEG): container finished" podID="8a62756c-45ea-4b59-9c74-5781f0ff4afc" containerID="7ed555867e236db6303c37400c97a3a4d77a5e784d7405773c82483dda71efe7" exitCode=0
Oct 30 13:31:05.503048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:05.502571    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6" event={"ID":"8a62756c-45ea-4b59-9c74-5781f0ff4afc","Type":"ContainerDied","Data":"7ed555867e236db6303c37400c97a3a4d77a5e784d7405773c82483dda71efe7"}
Oct 30 13:31:06.696439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.696405    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:06.820488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.820392    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-openshift-ca\") pod \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") "
Oct 30 13:31:06.820673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.820521    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a62756c-45ea-4b59-9c74-5781f0ff4afc-kube-api-access-z6xqp\" (UniqueName: \"kubernetes.io/projected/8a62756c-45ea-4b59-9c74-5781f0ff4afc-kube-api-access-z6xqp\") pod \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") "
Oct 30 13:31:06.820673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.820552    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-xks-ca\") pod \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\" (UID: \"8a62756c-45ea-4b59-9c74-5781f0ff4afc\") "
Oct 30 13:31:06.820870 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.820841    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-openshift-ca" pod "8a62756c-45ea-4b59-9c74-5781f0ff4afc" (UID: "8a62756c-45ea-4b59-9c74-5781f0ff4afc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:06.821123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.821099    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:31:06.822759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.822731    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-xks-ca" pod "8a62756c-45ea-4b59-9c74-5781f0ff4afc" (UID: "8a62756c-45ea-4b59-9c74-5781f0ff4afc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:31:06.822893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.822871    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a62756c-45ea-4b59-9c74-5781f0ff4afc-kube-api-access-z6xqp" pod "8a62756c-45ea-4b59-9c74-5781f0ff4afc" (UID: "8a62756c-45ea-4b59-9c74-5781f0ff4afc"). InnerVolumeSpecName "kube-api-access-z6xqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:06.922132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.922088    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6xqp\" (UniqueName: \"kubernetes.io/projected/8a62756c-45ea-4b59-9c74-5781f0ff4afc-kube-api-access-z6xqp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:31:06.922132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:06.922125    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a62756c-45ea-4b59-9c74-5781f0ff4afc-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:31:07.521810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:07.521781    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"
Oct 30 13:31:07.522019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:07.521821    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6" event={"ID":"8a62756c-45ea-4b59-9c74-5781f0ff4afc","Type":"ContainerDied","Data":"709e7d4a2828d4bc4433718fadd590accc593cd7e56275b3d56f89e958981d69"}
Oct 30 13:31:07.522019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:07.521875    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="709e7d4a2828d4bc4433718fadd590accc593cd7e56275b3d56f89e958981d69"
Oct 30 13:31:08.568337 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:31:08.568297    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:31:13.197225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.197181    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"]
Oct 30 13:31:13.197841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.197825    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8a62756c-45ea-4b59-9c74-5781f0ff4afc" containerName="revoke-keys"
Oct 30 13:31:13.197892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.197841    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a62756c-45ea-4b59-9c74-5781f0ff4afc" containerName="revoke-keys"
Oct 30 13:31:13.197950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.197941    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a62756c-45ea-4b59-9c74-5781f0ff4afc" containerName="revoke-keys"
Oct 30 13:31:13.201767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.201746    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.239435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.239388    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"]
Oct 30 13:31:13.250560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.250526    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.250749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.250578    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.250749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.250691    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.250886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.250781    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d662a085-9c77-4c7c-82e1-de6c7101715d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.250886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.250826    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.251012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.250918    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswgk\" (UniqueName: \"kubernetes.io/projected/d662a085-9c77-4c7c-82e1-de6c7101715d-kube-api-access-cswgk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.251012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.250944    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.251126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.251016    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.251126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.251086    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352328    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352410    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d662a085-9c77-4c7c-82e1-de6c7101715d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352449    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352518    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswgk\" (UniqueName: \"kubernetes.io/projected/d662a085-9c77-4c7c-82e1-de6c7101715d-kube-api-access-cswgk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352551    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352597    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352681    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352732    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352789    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.352945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.352810    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.353095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.353036    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.353419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.353393    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.353563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.353464    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d662a085-9c77-4c7c-82e1-de6c7101715d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.353563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.353477    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.355332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.355312    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.355510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.355487    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.380096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.380067    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswgk\" (UniqueName: \"kubernetes.io/projected/d662a085-9c77-4c7c-82e1-de6c7101715d-kube-api-access-cswgk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.407910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.407866    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d662a085-9c77-4c7c-82e1-de6c7101715d-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-czngj\" (UID: \"d662a085-9c77-4c7c-82e1-de6c7101715d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.523945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.523665    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:13.867963 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:31:13.867924    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd662a085_9c77_4c7c_82e1_de6c7101715d.slice/crio-144bc396f1ccaecdf8cf33586c7392f4163df30bc94d629fa47d4f1863ed72d4 WatchSource:0}: Error finding container 144bc396f1ccaecdf8cf33586c7392f4163df30bc94d629fa47d4f1863ed72d4: Status 404 returned error can't find the container with id 144bc396f1ccaecdf8cf33586c7392f4163df30bc94d629fa47d4f1863ed72d4
Oct 30 13:31:13.868125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.868031    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"]
Oct 30 13:31:13.871746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.871706    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:13.871887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.871790    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:13.871887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:13.871834    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:14.134290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:14.134029    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"]
Oct 30 13:31:14.172291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:14.172247    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4af9f1c-p2vks"]
Oct 30 13:31:14.601186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:14.601135    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" event={"ID":"d662a085-9c77-4c7c-82e1-de6c7101715d","Type":"ContainerStarted","Data":"52c5479bd3b988285a5c50e847c57385fa28b798f4df987398d5149fd68742ce"}
Oct 30 13:31:14.601679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:14.601239    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" event={"ID":"d662a085-9c77-4c7c-82e1-de6c7101715d","Type":"ContainerStarted","Data":"144bc396f1ccaecdf8cf33586c7392f4163df30bc94d629fa47d4f1863ed72d4"}
Oct 30 13:31:14.659786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:14.659321    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" podStartSLOduration=1.659304787 podStartE2EDuration="1.659304787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:14.657004475 +0000 UTC m=+3539.132724086" watchObservedRunningTime="2026-10-30 13:31:14.659304787 +0000 UTC m=+3539.135024394"
Oct 30 13:31:15.533876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:15.524590    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:15.533876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:15.533812    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-czngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 13:31:15.534171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:15.533891    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" podUID="d662a085-9c77-4c7c-82e1-de6c7101715d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 13:31:15.698851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:15.698809    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e3d750-c6a6-4f66-8926-877b792c0f9e" path="/var/lib/kubelet/pods/d0e3d750-c6a6-4f66-8926-877b792c0f9e/volumes"
Oct 30 13:31:15.920295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:15.917893    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"]
Oct 30 13:31:16.526642 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:16.526602    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-czngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 13:31:16.526892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:16.526684    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" podUID="d662a085-9c77-4c7c-82e1-de6c7101715d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 13:31:17.525012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.524973    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-czngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 13:31:17.525473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.525036    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" podUID="d662a085-9c77-4c7c-82e1-de6c7101715d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 13:31:17.794968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.794875    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"]
Oct 30 13:31:17.800671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.800645    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:17.815283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.815252    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-99f4a4\"/\"openshift-service-ca.crt\""
Oct 30 13:31:17.817882 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.817857    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"]
Oct 30 13:31:17.827016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.826973    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-99f4a4\"/\"kube-root-ca.crt\""
Oct 30 13:31:17.827166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.826974    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-99f4a4\"/\"default-dockercfg-csst8\""
Oct 30 13:31:17.827166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.826974    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-99f4a4\"/\"e2e-usage-sim-ee9e02-cert\""
Oct 30 13:31:17.850938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.850813    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-tls\") pod \"e2e-usage-sim-ee9e02-d5fd59d59-t2z4p\" (UID: \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\") " pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:17.851302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.851275    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6c6l\" (UniqueName: \"kubernetes.io/projected/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-kube-api-access-c6c6l\") pod \"e2e-usage-sim-ee9e02-d5fd59d59-t2z4p\" (UID: \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\") " pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:17.953599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.953552    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-tls\") pod \"e2e-usage-sim-ee9e02-d5fd59d59-t2z4p\" (UID: \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\") " pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:17.954457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.954427    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6c6l\" (UniqueName: \"kubernetes.io/projected/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-kube-api-access-c6c6l\") pod \"e2e-usage-sim-ee9e02-d5fd59d59-t2z4p\" (UID: \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\") " pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:17.982706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.982667    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-tls\") pod \"e2e-usage-sim-ee9e02-d5fd59d59-t2z4p\" (UID: \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\") " pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:17.987937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:17.987898    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6c6l\" (UniqueName: \"kubernetes.io/projected/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-kube-api-access-c6c6l\") pod \"e2e-usage-sim-ee9e02-d5fd59d59-t2z4p\" (UID: \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\") " pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:18.114950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.114856    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:18.492845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.491876    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"]
Oct 30 13:31:18.492845 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:31:18.492452    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e06be5f_7e1e_4060_9e9a_5d5a893ebfe6.slice/crio-12abd604760d1b43da23f0a5c52d7381b5a697ff93688fc00380210b65f78f97 WatchSource:0}: Error finding container 12abd604760d1b43da23f0a5c52d7381b5a697ff93688fc00380210b65f78f97: Status 404 returned error can't find the container with id 12abd604760d1b43da23f0a5c52d7381b5a697ff93688fc00380210b65f78f97
Oct 30 13:31:18.524492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.524463    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-czngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 13:31:18.524644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.524513    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" podUID="d662a085-9c77-4c7c-82e1-de6c7101715d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 13:31:18.732097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.732055    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p" event={"ID":"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6","Type":"ContainerStarted","Data":"1115374a94adf88c739df1f702769afd6ff9aca10752622df1c07fac4d875c0a"}
Oct 30 13:31:18.732594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.732119    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:18.732594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.732134    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p" event={"ID":"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6","Type":"ContainerStarted","Data":"12abd604760d1b43da23f0a5c52d7381b5a697ff93688fc00380210b65f78f97"}
Oct 30 13:31:18.738368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.738338    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:31:18.816366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:18.816279    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p" podStartSLOduration=1.816258941 podStartE2EDuration="1.816258941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:18.776456269 +0000 UTC m=+3543.252175882" watchObservedRunningTime="2026-10-30 13:31:18.816258941 +0000 UTC m=+3543.291978552"
Oct 30 13:31:19.524333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:19.524292    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-czngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 13:31:19.524528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:19.524355    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" podUID="d662a085-9c77-4c7c-82e1-de6c7101715d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 13:31:20.390127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:20.390084    4993 scope.go:122] "RemoveContainer" containerID="9985cf87415434cd886e248a0976d027209c768d4be7cf299812d79db2c01fa7"
Oct 30 13:31:20.525005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:20.524957    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-czngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 13:31:20.525076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:20.525016    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" podUID="d662a085-9c77-4c7c-82e1-de6c7101715d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 13:31:20.926229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:20.925281    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq" podUID="aa35769e-4296-4984-9033-478847869c19" containerName="payload-processing" containerID="cri-o://dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc" gracePeriod=25
Oct 30 13:31:21.258720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.258685    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:31:21.340639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.340578    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-provider-credentials\") pod \"aa35769e-4296-4984-9033-478847869c19\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") "
Oct 30 13:31:21.340829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.340790    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-kube-api-access-dgxwf\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-kube-api-access-dgxwf\") pod \"aa35769e-4296-4984-9033-478847869c19\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") "
Oct 30 13:31:21.340931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.340865    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-routing-overlay-volume\") pod \"aa35769e-4296-4984-9033-478847869c19\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") "
Oct 30 13:31:21.340931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.340907    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-plugins-config-volume\") pod \"aa35769e-4296-4984-9033-478847869c19\" (UID: \"aa35769e-4296-4984-9033-478847869c19\") "
Oct 30 13:31:21.341471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.341271    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-routing-overlay-volume" pod "aa35769e-4296-4984-9033-478847869c19" (UID: "aa35769e-4296-4984-9033-478847869c19"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:21.341615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.341547    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-plugins-config-volume" pod "aa35769e-4296-4984-9033-478847869c19" (UID: "aa35769e-4296-4984-9033-478847869c19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:21.341674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.341613    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:31:21.343811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.343772    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-provider-credentials" pod "aa35769e-4296-4984-9033-478847869c19" (UID: "aa35769e-4296-4984-9033-478847869c19"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:21.345685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.345656    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-kube-api-access-dgxwf" pod "aa35769e-4296-4984-9033-478847869c19" (UID: "aa35769e-4296-4984-9033-478847869c19"). InnerVolumeSpecName "kube-api-access-dgxwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:21.442621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.442584    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa35769e-4296-4984-9033-478847869c19-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:31:21.442621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.442617    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:31:21.442621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.442627    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgxwf\" (UniqueName: \"kubernetes.io/projected/aa35769e-4296-4984-9033-478847869c19-kube-api-access-dgxwf\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:31:21.524938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.524893    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-czngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 13:31:21.525168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.524964    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj" podUID="d662a085-9c77-4c7c-82e1-de6c7101715d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 13:31:21.760368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.760328    4993 generic.go:356] "Generic (PLEG): container finished" podID="aa35769e-4296-4984-9033-478847869c19" containerID="dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc" exitCode=0
Oct 30 13:31:21.760555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.760380    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq" event={"ID":"aa35769e-4296-4984-9033-478847869c19","Type":"ContainerDied","Data":"dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc"}
Oct 30 13:31:21.760555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.760390    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"
Oct 30 13:31:21.760555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.760443    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq" event={"ID":"aa35769e-4296-4984-9033-478847869c19","Type":"ContainerDied","Data":"1a1f604792304891189f2a695a01205140e9a6f71fe7f26de77f85ac68923877"}
Oct 30 13:31:21.760555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.760478    4993 scope.go:122] "RemoveContainer" containerID="dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc"
Oct 30 13:31:21.774838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.774810    4993 scope.go:122] "RemoveContainer" containerID="dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc"
Oct 30 13:31:21.775250 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:31:21.775176    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc\": container with ID starting with dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc not found: ID does not exist" containerID="dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc"
Oct 30 13:31:21.775365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.775247    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc"} err="failed to get container status \"dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc\": rpc error: code = NotFound desc = could not find container \"dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc\": container with ID starting with dd8a72d868a731f5b14252f4ede8e104434f15dce11fef8c59f655371de7affc not found: ID does not exist"
Oct 30 13:31:21.823987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.823951    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"]
Oct 30 13:31:21.836692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:21.836657    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-63499e/payload-processing-external-model-e2e-extmod-a-63499e-6c55psmdq"]
Oct 30 13:31:23.425991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:23.425955    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:23.426576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:23.426544    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:23.427614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:23.427591    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-czngj"
Oct 30 13:31:23.695449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:23.695352    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa35769e-4296-4984-9033-478847869c19" path="/var/lib/kubelet/pods/aa35769e-4296-4984-9033-478847869c19/volumes"
Oct 30 13:31:25.081259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:25.081202    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:31:30.032523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:30.032487    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"]
Oct 30 13:31:30.039512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:30.039466    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf29d12-k5hzs"]
Oct 30 13:31:31.496516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.496471    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"]
Oct 30 13:31:31.497478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.497456    4993 cpu_manager.go:414] "containerMap: removing container" podUID="aa35769e-4296-4984-9033-478847869c19" containerName="payload-processing"
Oct 30 13:31:31.497559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.497479    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa35769e-4296-4984-9033-478847869c19" containerName="payload-processing"
Oct 30 13:31:31.497649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.497636    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa35769e-4296-4984-9033-478847869c19" containerName="payload-processing"
Oct 30 13:31:31.503412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.503380    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.533393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.533359    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"agc-cred-32aceaef5cf0\""
Oct 30 13:31:31.533613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.533424    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"openshift-service-ca.crt\""
Oct 30 13:31:31.533693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.533445    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"payload-processing-external-model-e2e--65402e9a-dockercfg-nl6ct\""
Oct 30 13:31:31.533693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.533448    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0d9115\""
Oct 30 13:31:31.533778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.533735    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"kube-root-ca.crt\""
Oct 30 13:31:31.536261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.536242    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"routing-overlay\""
Oct 30 13:31:31.539413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.539378    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"]
Oct 30 13:31:31.639563 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.639518    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.639779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.639651    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lbb\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-kube-api-access-f9lbb\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.639779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.639673    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.639967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.639941    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.735228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.734838    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e52616-6f4a-4c50-833c-dd3e4ebd4208" path="/var/lib/kubelet/pods/17e52616-6f4a-4c50-833c-dd3e4ebd4208/volumes"
Oct 30 13:31:31.741280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.741240    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.741635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.741606    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.741885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.741865    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lbb\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-kube-api-access-f9lbb\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.742019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.742001    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.750104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.750040    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.751003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.750979    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.766228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.751946    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.804081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.804029    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lbb\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-kube-api-access-f9lbb\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:31.820522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:31.820484    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:32.320279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:32.312796    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"]
Oct 30 13:31:33.087018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.086969    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd" event={"ID":"7757eb1e-51c8-4498-9bed-ada982488167","Type":"ContainerStarted","Data":"94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49"}
Oct 30 13:31:33.087554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.087045    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd" event={"ID":"7757eb1e-51c8-4498-9bed-ada982488167","Type":"ContainerStarted","Data":"716fdac4c0585969204180335bd40c5d02265a7d4834bb64b90a44ae3d1d5fbb"}
Oct 30 13:31:33.087554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.087076    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:33.131200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.131134    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd" podStartSLOduration=2.131114457 podStartE2EDuration="2.131114457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:33.127795311 +0000 UTC m=+3557.603514924" watchObservedRunningTime="2026-10-30 13:31:33.131114457 +0000 UTC m=+3557.606834070"
Oct 30 13:31:33.562531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.562495    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"]
Oct 30 13:31:33.567254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.566930    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.582249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.582200    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 13:31:33.582460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.582421    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 13:31:33.582460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.582427    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 13:31:33.582811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.582773    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-88e3259e1834\""
Oct 30 13:31:33.594280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.594243    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 13:31:33.594461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.594290    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-fcnjv\""
Oct 30 13:31:33.599572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.599536    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"]
Oct 30 13:31:33.771244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.757085    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-plugins-config-volume\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.771244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.757181    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvjx2\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-kube-api-access-zvjx2\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.771244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.757310    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-provider-credentials\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.771244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.757540    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-routing-overlay-volume\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.888327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.885822    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-routing-overlay-volume\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.888327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.885952    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-plugins-config-volume\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.888327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.886035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvjx2\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-kube-api-access-zvjx2\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.888327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.886109    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-provider-credentials\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.899259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.897284    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-plugins-config-volume\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.899259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.897879    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-routing-overlay-volume\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.944229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.936473    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-provider-credentials\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:33.947384 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:33.947169    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvjx2\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-kube-api-access-zvjx2\") pod \"payload-processing-external-model-5cc5856c6b-765qs\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") " pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:34.182954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:34.182849    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:34.514243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:34.514193    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"]
Oct 30 13:31:34.514987 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:31:34.514964    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcff096_8adf_48be_ad7d_6841360d86b1.slice/crio-99158ab72afdac42dc1d243207ada352b2a62f3e344446312cd5eab333a90428 WatchSource:0}: Error finding container 99158ab72afdac42dc1d243207ada352b2a62f3e344446312cd5eab333a90428: Status 404 returned error can't find the container with id 99158ab72afdac42dc1d243207ada352b2a62f3e344446312cd5eab333a90428
Oct 30 13:31:35.111359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:35.111308    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs" event={"ID":"efcff096-8adf-48be-ad7d-6841360d86b1","Type":"ContainerStarted","Data":"fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a"}
Oct 30 13:31:35.111557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:35.111374    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs" event={"ID":"efcff096-8adf-48be-ad7d-6841360d86b1","Type":"ContainerStarted","Data":"99158ab72afdac42dc1d243207ada352b2a62f3e344446312cd5eab333a90428"}
Oct 30 13:31:35.111557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:35.111490    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:31:35.154079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:35.153992    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs" podStartSLOduration=2.153973294 podStartE2EDuration="2.153973294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:35.151643201 +0000 UTC m=+3559.627362811" watchObservedRunningTime="2026-10-30 13:31:35.153973294 +0000 UTC m=+3559.629692905"
Oct 30 13:31:38.032444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:38.032405    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"]
Oct 30 13:31:38.044540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:38.044501    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae083fe5-6t9r8"]
Oct 30 13:31:38.570187 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:31:38.570146    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:31:39.692398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:39.692354    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a9adda-d603-436e-b5fd-a3c6523268d0" path="/var/lib/kubelet/pods/d3a9adda-d603-436e-b5fd-a3c6523268d0/volumes"
Oct 30 13:31:44.102574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:44.102533    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:31:46.124040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:31:46.123986    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:32:04.553227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:04.552348    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:04.825698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:04.824348    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:32:04.956423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:04.956355    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:05.135292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:05.135146    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:05.208361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:05.208307    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:05.555370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:05.555312    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:05.556403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:05.556375    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 13:32:06.097232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.094715    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 13:32:06.558182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.558137    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"]
Oct 30 13:32:06.562680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.562633    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.582019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.581978    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:32:06.582317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.582292    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:32:06.582485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.582463    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:32:06.582600 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.582548    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:32:06.590766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.590733    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"]
Oct 30 13:32:06.644395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.644340    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.644606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.644559    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skr9l\" (UniqueName: \"kubernetes.io/projected/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-kube-api-access-skr9l\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.644723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.644702    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.746571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.746529    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.746771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.746683    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skr9l\" (UniqueName: \"kubernetes.io/projected/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-kube-api-access-skr9l\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.746842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.746795    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.747547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.747512    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.749260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.749231    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.770755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.770699    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skr9l\" (UniqueName: \"kubernetes.io/projected/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-kube-api-access-skr9l\") pod \"maas-api-revoke-sub-e75dadf4-crszm\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:06.882132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:06.882030    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:07.197757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:07.197724    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"]
Oct 30 13:32:07.200641 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:32:07.200609    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f749cc_c785_4f0b_9a5d_2e09a27d2b4b.slice/crio-d6f97e7f4990a1c9a83fea1127aa058e90eaa884c7a2bdcd805ca2ec47f00e37 WatchSource:0}: Error finding container d6f97e7f4990a1c9a83fea1127aa058e90eaa884c7a2bdcd805ca2ec47f00e37: Status 404 returned error can't find the container with id d6f97e7f4990a1c9a83fea1127aa058e90eaa884c7a2bdcd805ca2ec47f00e37
Oct 30 13:32:07.556291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:07.556247    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm" event={"ID":"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b","Type":"ContainerStarted","Data":"d6f97e7f4990a1c9a83fea1127aa058e90eaa884c7a2bdcd805ca2ec47f00e37"}
Oct 30 13:32:08.569495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:08.569450    4993 generic.go:356] "Generic (PLEG): container finished" podID="27f749cc-c785-4f0b-9a5d-2e09a27d2b4b" containerID="cc3af79210815a5b001772feff1805af27492b330700913ca5a06400e9fd7b4f" exitCode=0
Oct 30 13:32:08.570027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:08.569570    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm" event={"ID":"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b","Type":"ContainerDied","Data":"cc3af79210815a5b001772feff1805af27492b330700913ca5a06400e9fd7b4f"}
Oct 30 13:32:08.573093 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:32:08.573049    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:32:09.768287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:09.768251    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:09.894672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:09.894577    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-kube-api-access-skr9l\" (UniqueName: \"kubernetes.io/projected/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-kube-api-access-skr9l\") pod \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") "
Oct 30 13:32:09.894869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:09.894744    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-xks-ca\") pod \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") "
Oct 30 13:32:09.894869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:09.894781    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-openshift-ca\") pod \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\" (UID: \"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b\") "
Oct 30 13:32:09.895245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:09.895194    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-openshift-ca" pod "27f749cc-c785-4f0b-9a5d-2e09a27d2b4b" (UID: "27f749cc-c785-4f0b-9a5d-2e09a27d2b4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:09.895554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:09.895531    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:09.896838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:09.896803    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-kube-api-access-skr9l" pod "27f749cc-c785-4f0b-9a5d-2e09a27d2b4b" (UID: "27f749cc-c785-4f0b-9a5d-2e09a27d2b4b"). InnerVolumeSpecName "kube-api-access-skr9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:09.897107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:09.897085    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-xks-ca" pod "27f749cc-c785-4f0b-9a5d-2e09a27d2b4b" (UID: "27f749cc-c785-4f0b-9a5d-2e09a27d2b4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:32:10.025722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:10.025187    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:10.025722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:10.025246    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skr9l\" (UniqueName: \"kubernetes.io/projected/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b-kube-api-access-skr9l\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:10.591656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:10.591621    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"
Oct 30 13:32:10.591864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:10.591651    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm" event={"ID":"27f749cc-c785-4f0b-9a5d-2e09a27d2b4b","Type":"ContainerDied","Data":"d6f97e7f4990a1c9a83fea1127aa058e90eaa884c7a2bdcd805ca2ec47f00e37"}
Oct 30 13:32:10.591864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:10.591710    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6f97e7f4990a1c9a83fea1127aa058e90eaa884c7a2bdcd805ca2ec47f00e37"
Oct 30 13:32:16.347580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:16.347550    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:32:16.380564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:16.380532    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:32:17.883457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:17.883403    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"]
Oct 30 13:32:20.590541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.590506    4993 scope.go:122] "RemoveContainer" containerID="b14e9e71a6bb4b9a7f576b3bdd66b7e335c299af5f4fa986331c40aaac8f47ec"
Oct 30 13:32:20.652651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.652618    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"]
Oct 30 13:32:20.653821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.653795    4993 cpu_manager.go:414] "containerMap: removing container" podUID="27f749cc-c785-4f0b-9a5d-2e09a27d2b4b" containerName="revoke-keys"
Oct 30 13:32:20.653986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.653974    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f749cc-c785-4f0b-9a5d-2e09a27d2b4b" containerName="revoke-keys"
Oct 30 13:32:20.655480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.655456    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f749cc-c785-4f0b-9a5d-2e09a27d2b4b" containerName="revoke-keys"
Oct 30 13:32:20.664226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.660188    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.692406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.689598    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d41320-openshift-default-dockercfg-vgtdt\""
Oct 30 13:32:20.730907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.730863    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"]
Oct 30 13:32:20.818263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.810192    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-envoy\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.818263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.810340    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-token\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.818263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.810370    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istiod-ca-cert\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.818263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.810529    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-credential-socket\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.818263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.810579    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-socket\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.852534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.845417    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-certs\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.852534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.845527    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-podinfo\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.852534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.845596    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxhfw\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-kube-api-access-gxhfw\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.852534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.845685    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-data\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.963244    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-envoy\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.963900    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-token\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.964321    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istiod-ca-cert\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.964470    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-credential-socket\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.964518    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-socket\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.964640    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-certs\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.964689    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-podinfo\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.964735    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhfw\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-kube-api-access-gxhfw\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.964875 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.964810    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-data\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.971246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.965319    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-data\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.971246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.965958    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istiod-ca-cert\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.971246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.966190    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-credential-socket\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.979229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.976699    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-socket\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:20.979229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.977848    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-certs\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:21.002242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:20.999174    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-podinfo\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:21.012242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:21.009387    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-token\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:21.012657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:21.012392    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhfw\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-kube-api-access-gxhfw\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:21.021238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:21.017504    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-envoy\") pod \"e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") " pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:21.284996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:21.284656    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:21.706719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:21.706683    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"]
Oct 30 13:32:21.708439 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:32:21.708411    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c80a7fc_a3fa_46b8_8134_2bca5b383a6a.slice/crio-228a16f075df8d101f46379c779e52af8e5e7297b82aa33a842c42f3bb0d47e6 WatchSource:0}: Error finding container 228a16f075df8d101f46379c779e52af8e5e7297b82aa33a842c42f3bb0d47e6: Status 404 returned error can't find the container with id 228a16f075df8d101f46379c779e52af8e5e7297b82aa33a842c42f3bb0d47e6
Oct 30 13:32:21.711519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:21.711477    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:32:21.711645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:21.711555    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:32:21.711645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:21.711591    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:32:22.724286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:22.724238    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" event={"ID":"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a","Type":"ContainerStarted","Data":"6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b"}
Oct 30 13:32:22.724726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:22.724342    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" event={"ID":"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a","Type":"ContainerStarted","Data":"228a16f075df8d101f46379c779e52af8e5e7297b82aa33a842c42f3bb0d47e6"}
Oct 30 13:32:22.791331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:22.791251    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podStartSLOduration=2.791226235 podStartE2EDuration="2.791226235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:22.784730077 +0000 UTC m=+3607.260449686" watchObservedRunningTime="2026-10-30 13:32:22.791226235 +0000 UTC m=+3607.266945856"
Oct 30 13:32:22.884351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:22.884271    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs" podUID="efcff096-8adf-48be-ad7d-6841360d86b1" containerName="payload-processing" containerID="cri-o://fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a" gracePeriod=25
Oct 30 13:32:23.286725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.286691    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:23.293808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.293711    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:32:23.294017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.293979    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:23.369617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.368636    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-kube-api-access-zvjx2\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-kube-api-access-zvjx2\") pod \"efcff096-8adf-48be-ad7d-6841360d86b1\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") "
Oct 30 13:32:23.369617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.368969    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-plugins-config-volume\") pod \"efcff096-8adf-48be-ad7d-6841360d86b1\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") "
Oct 30 13:32:23.369617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.369046    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-provider-credentials\") pod \"efcff096-8adf-48be-ad7d-6841360d86b1\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") "
Oct 30 13:32:23.369617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.369092    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-routing-overlay-volume\") pod \"efcff096-8adf-48be-ad7d-6841360d86b1\" (UID: \"efcff096-8adf-48be-ad7d-6841360d86b1\") "
Oct 30 13:32:23.370520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.370479    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-routing-overlay-volume" pod "efcff096-8adf-48be-ad7d-6841360d86b1" (UID: "efcff096-8adf-48be-ad7d-6841360d86b1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:23.373814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.373766    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-plugins-config-volume" pod "efcff096-8adf-48be-ad7d-6841360d86b1" (UID: "efcff096-8adf-48be-ad7d-6841360d86b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:23.375436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.375380    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-kube-api-access-zvjx2" pod "efcff096-8adf-48be-ad7d-6841360d86b1" (UID: "efcff096-8adf-48be-ad7d-6841360d86b1"). InnerVolumeSpecName "kube-api-access-zvjx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:23.376932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.376886    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-provider-credentials" pod "efcff096-8adf-48be-ad7d-6841360d86b1" (UID: "efcff096-8adf-48be-ad7d-6841360d86b1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:23.475272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.471878    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:23.475272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.471916    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:23.475272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.471932    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efcff096-8adf-48be-ad7d-6841360d86b1-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:23.475272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.471945    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvjx2\" (UniqueName: \"kubernetes.io/projected/efcff096-8adf-48be-ad7d-6841360d86b1-kube-api-access-zvjx2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:23.742718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.742665    4993 generic.go:356] "Generic (PLEG): container finished" podID="efcff096-8adf-48be-ad7d-6841360d86b1" containerID="fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a" exitCode=0
Oct 30 13:32:23.744429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.744355    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"
Oct 30 13:32:23.745049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.745023    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs" event={"ID":"efcff096-8adf-48be-ad7d-6841360d86b1","Type":"ContainerDied","Data":"fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a"}
Oct 30 13:32:23.745161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.745076    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs" event={"ID":"efcff096-8adf-48be-ad7d-6841360d86b1","Type":"ContainerDied","Data":"99158ab72afdac42dc1d243207ada352b2a62f3e344446312cd5eab333a90428"}
Oct 30 13:32:23.745161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.745108    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:23.745161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.745140    4993 scope.go:122] "RemoveContainer" containerID="fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a"
Oct 30 13:32:23.747553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.747528    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:32:23.761334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.761298    4993 scope.go:122] "RemoveContainer" containerID="fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a"
Oct 30 13:32:23.761762 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:32:23.761731    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a\": container with ID starting with fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a not found: ID does not exist" containerID="fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a"
Oct 30 13:32:23.761842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.761790    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a"} err="failed to get container status \"fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a\": rpc error: code = NotFound desc = could not find container \"fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a\": container with ID starting with fe9f6241af1f2372b2dedc0da6a920cb452ebbc47713f4aa966bea2b72850e0a not found: ID does not exist"
Oct 30 13:32:23.839331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.839280    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"]
Oct 30 13:32:23.856930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:23.856889    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc5856c6b-765qs"]
Oct 30 13:32:25.767982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:25.767941    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efcff096-8adf-48be-ad7d-6841360d86b1" path="/var/lib/kubelet/pods/efcff096-8adf-48be-ad7d-6841360d86b1/volumes"
Oct 30 13:32:38.583049 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:32:38.582999    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:32:49.419240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:49.413361    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:49.449240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:49.449001    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:50.426231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:50.425304    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:50.426231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:50.426131    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:32:50.527232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:50.509292    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:50.909256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:50.820329    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:50.909256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:50.857645    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:32:51.099230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:51.099105    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 13:32:51.145360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:51.145310    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:51.205933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:51.205834    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:52.276794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.276756    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"]
Oct 30 13:32:52.277603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.277579    4993 cpu_manager.go:414] "containerMap: removing container" podUID="efcff096-8adf-48be-ad7d-6841360d86b1" containerName="payload-processing"
Oct 30 13:32:52.277717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.277603    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efcff096-8adf-48be-ad7d-6841360d86b1" containerName="payload-processing"
Oct 30 13:32:52.277802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.277787    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="efcff096-8adf-48be-ad7d-6841360d86b1" containerName="payload-processing"
Oct 30 13:32:52.281639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.281618    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.303810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.303776    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"]
Oct 30 13:32:52.309109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.309081    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:32:52.309320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.309112    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:32:52.309320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.309118    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:32:52.309474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.309179    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:32:52.384582 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.384544    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.384810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.384615    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtjp\" (UniqueName: \"kubernetes.io/projected/fe9009be-b913-42d6-bd52-2a8d3c57dd40-kube-api-access-cvtjp\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.384810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.384704    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.488232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.486255    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.488232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.486600    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.488232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.486683    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtjp\" (UniqueName: \"kubernetes.io/projected/fe9009be-b913-42d6-bd52-2a8d3c57dd40-kube-api-access-cvtjp\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.488232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.488064    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.519527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.518061    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.525070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.523971    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtjp\" (UniqueName: \"kubernetes.io/projected/fe9009be-b913-42d6-bd52-2a8d3c57dd40-kube-api-access-cvtjp\") pod \"maas-api-revoke-sub-dd4a7799-kgxrh\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.601041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.600957    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:52.918358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:52.918326    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"]
Oct 30 13:32:52.927837 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:32:52.926485    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9009be_b913_42d6_bd52_2a8d3c57dd40.slice/crio-845d9c0b730c0b66d16cb77aa7bbc42e79d3148d6d5b46250ccace29544dfd81 WatchSource:0}: Error finding container 845d9c0b730c0b66d16cb77aa7bbc42e79d3148d6d5b46250ccace29544dfd81: Status 404 returned error can't find the container with id 845d9c0b730c0b66d16cb77aa7bbc42e79d3148d6d5b46250ccace29544dfd81
Oct 30 13:32:53.353228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:53.344574    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh" event={"ID":"fe9009be-b913-42d6-bd52-2a8d3c57dd40","Type":"ContainerStarted","Data":"845d9c0b730c0b66d16cb77aa7bbc42e79d3148d6d5b46250ccace29544dfd81"}
Oct 30 13:32:54.355313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:54.355276    4993 generic.go:356] "Generic (PLEG): container finished" podID="fe9009be-b913-42d6-bd52-2a8d3c57dd40" containerID="c82b84cdc81d81301a437cc504e192596906e8e3f6672161f987d3360fe9358b" exitCode=0
Oct 30 13:32:54.355746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:54.355346    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh" event={"ID":"fe9009be-b913-42d6-bd52-2a8d3c57dd40","Type":"ContainerDied","Data":"c82b84cdc81d81301a437cc504e192596906e8e3f6672161f987d3360fe9358b"}
Oct 30 13:32:55.033130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.033086    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"]
Oct 30 13:32:55.037927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.037891    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580313b2-8w6m8"]
Oct 30 13:32:55.566137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.566097    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:55.680257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.680137    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-xks-ca\") pod \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") "
Oct 30 13:32:55.680424 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.680369    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9009be-b913-42d6-bd52-2a8d3c57dd40-kube-api-access-cvtjp\" (UniqueName: \"kubernetes.io/projected/fe9009be-b913-42d6-bd52-2a8d3c57dd40-kube-api-access-cvtjp\") pod \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") "
Oct 30 13:32:55.680470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.680437    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-openshift-ca\") pod \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\" (UID: \"fe9009be-b913-42d6-bd52-2a8d3c57dd40\") "
Oct 30 13:32:55.680994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.680959    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-openshift-ca" pod "fe9009be-b913-42d6-bd52-2a8d3c57dd40" (UID: "fe9009be-b913-42d6-bd52-2a8d3c57dd40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:55.682418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.682395    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-xks-ca" pod "fe9009be-b913-42d6-bd52-2a8d3c57dd40" (UID: "fe9009be-b913-42d6-bd52-2a8d3c57dd40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:32:55.683008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.682991    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9009be-b913-42d6-bd52-2a8d3c57dd40-kube-api-access-cvtjp" pod "fe9009be-b913-42d6-bd52-2a8d3c57dd40" (UID: "fe9009be-b913-42d6-bd52-2a8d3c57dd40"). InnerVolumeSpecName "kube-api-access-cvtjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:55.693758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.693722    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87290992-a709-4f80-87ca-76311600e068" path="/var/lib/kubelet/pods/87290992-a709-4f80-87ca-76311600e068/volumes"
Oct 30 13:32:55.782139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.782102    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:55.782139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.782139    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvtjp\" (UniqueName: \"kubernetes.io/projected/fe9009be-b913-42d6-bd52-2a8d3c57dd40-kube-api-access-cvtjp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:55.782413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:55.782154    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9009be-b913-42d6-bd52-2a8d3c57dd40-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:32:56.379202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:56.379116    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"
Oct 30 13:32:56.379202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:56.379135    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh" event={"ID":"fe9009be-b913-42d6-bd52-2a8d3c57dd40","Type":"ContainerDied","Data":"845d9c0b730c0b66d16cb77aa7bbc42e79d3148d6d5b46250ccace29544dfd81"}
Oct 30 13:32:56.379447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:32:56.379234    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="845d9c0b730c0b66d16cb77aa7bbc42e79d3148d6d5b46250ccace29544dfd81"
Oct 30 13:33:03.574827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:03.574778    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"]
Oct 30 13:33:04.285978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.285948    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"]
Oct 30 13:33:04.286879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.286845    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fe9009be-b913-42d6-bd52-2a8d3c57dd40" containerName="revoke-keys"
Oct 30 13:33:04.287016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.287004    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9009be-b913-42d6-bd52-2a8d3c57dd40" containerName="revoke-keys"
Oct 30 13:33:04.287280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.287265    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9009be-b913-42d6-bd52-2a8d3c57dd40" containerName="revoke-keys"
Oct 30 13:33:04.291345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.291326    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.312958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.312920    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"]
Oct 30 13:33:04.318126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.318090    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:33:04.318308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.318197    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:33:04.318308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.318262    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:33:04.318427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.318330    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:33:04.344345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.344300    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zt4\" (UniqueName: \"kubernetes.io/projected/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-kube-api-access-g9zt4\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.344567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.344364    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.344567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.344551    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.446025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.445989    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.446261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.446187    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.446359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.446341    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zt4\" (UniqueName: \"kubernetes.io/projected/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-kube-api-access-g9zt4\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.446881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.446853    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.448720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.448693    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.469693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.469663    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zt4\" (UniqueName: \"kubernetes.io/projected/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-kube-api-access-g9zt4\") pod \"maas-api-revoke-sub-f0edbdcb-l8dr4\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.607713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.607571    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:04.941422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:04.941391    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"]
Oct 30 13:33:04.943726 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:33:04.943688    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode388c5d0_9415_4eae_9b73_ebc02fcb8e13.slice/crio-6bc61aff7fd5f8d535eec1e276f8d1712f506f32264da27985441db33698a1b5 WatchSource:0}: Error finding container 6bc61aff7fd5f8d535eec1e276f8d1712f506f32264da27985441db33698a1b5: Status 404 returned error can't find the container with id 6bc61aff7fd5f8d535eec1e276f8d1712f506f32264da27985441db33698a1b5
Oct 30 13:33:05.459025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:05.458981    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4" event={"ID":"e388c5d0-9415-4eae-9b73-ebc02fcb8e13","Type":"ContainerStarted","Data":"765f8d9055680f9f00ecd92a168fcef9f87aabbe3dfe5f2920875f47f4580a23"}
Oct 30 13:33:05.459240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:05.459049    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4" event={"ID":"e388c5d0-9415-4eae-9b73-ebc02fcb8e13","Type":"ContainerStarted","Data":"6bc61aff7fd5f8d535eec1e276f8d1712f506f32264da27985441db33698a1b5"}
Oct 30 13:33:05.504366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:05.504192    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4" podStartSLOduration=1.504173325 podStartE2EDuration="1.504173325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:05.501266815 +0000 UTC m=+3649.976986425" watchObservedRunningTime="2026-10-30 13:33:05.504173325 +0000 UTC m=+3649.979892936"
Oct 30 13:33:06.041004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:06.038849    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"]
Oct 30 13:33:06.049515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:06.049469    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51ec470-82bgl"]
Oct 30 13:33:06.469349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:06.468804    4993 generic.go:356] "Generic (PLEG): container finished" podID="e388c5d0-9415-4eae-9b73-ebc02fcb8e13" containerID="765f8d9055680f9f00ecd92a168fcef9f87aabbe3dfe5f2920875f47f4580a23" exitCode=0
Oct 30 13:33:06.469349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:06.469032    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4" event={"ID":"e388c5d0-9415-4eae-9b73-ebc02fcb8e13","Type":"ContainerDied","Data":"765f8d9055680f9f00ecd92a168fcef9f87aabbe3dfe5f2920875f47f4580a23"}
Oct 30 13:33:07.677850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.677816    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:07.695444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.695401    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf277479-f049-4e98-84a2-3e4da9846796" path="/var/lib/kubelet/pods/bf277479-f049-4e98-84a2-3e4da9846796/volumes"
Oct 30 13:33:07.695965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.695929    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-kube-api-access-g9zt4\" (UniqueName: \"kubernetes.io/projected/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-kube-api-access-g9zt4\") pod \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") "
Oct 30 13:33:07.696122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.696107    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-openshift-ca\") pod \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") "
Oct 30 13:33:07.696349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.696328    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-xks-ca\") pod \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\" (UID: \"e388c5d0-9415-4eae-9b73-ebc02fcb8e13\") "
Oct 30 13:33:07.696664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.696635    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-openshift-ca" pod "e388c5d0-9415-4eae-9b73-ebc02fcb8e13" (UID: "e388c5d0-9415-4eae-9b73-ebc02fcb8e13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:07.697094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.697063    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:07.698845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.698806    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-kube-api-access-g9zt4" pod "e388c5d0-9415-4eae-9b73-ebc02fcb8e13" (UID: "e388c5d0-9415-4eae-9b73-ebc02fcb8e13"). InnerVolumeSpecName "kube-api-access-g9zt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:07.698953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.698848    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-xks-ca" pod "e388c5d0-9415-4eae-9b73-ebc02fcb8e13" (UID: "e388c5d0-9415-4eae-9b73-ebc02fcb8e13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:33:07.798802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.798711    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:07.798802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:07.798753    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zt4\" (UniqueName: \"kubernetes.io/projected/e388c5d0-9415-4eae-9b73-ebc02fcb8e13-kube-api-access-g9zt4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:08.489648 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:08.489615    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"
Oct 30 13:33:08.489648 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:08.489639    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4" event={"ID":"e388c5d0-9415-4eae-9b73-ebc02fcb8e13","Type":"ContainerDied","Data":"6bc61aff7fd5f8d535eec1e276f8d1712f506f32264da27985441db33698a1b5"}
Oct 30 13:33:08.489937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:08.489689    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc61aff7fd5f8d535eec1e276f8d1712f506f32264da27985441db33698a1b5"
Oct 30 13:33:08.573352 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:33:08.573304    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:33:08.575516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:08.575483    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd" podUID="7757eb1e-51c8-4498-9bed-ada982488167" containerName="payload-processing" containerID="cri-o://94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49" gracePeriod=25
Oct 30 13:33:08.924867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:08.924836    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:33:09.015902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.015858    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-kube-api-access-f9lbb\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-kube-api-access-f9lbb\") pod \"7757eb1e-51c8-4498-9bed-ada982488167\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") "
Oct 30 13:33:09.016138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.015937    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-routing-overlay-volume\") pod \"7757eb1e-51c8-4498-9bed-ada982488167\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") "
Oct 30 13:33:09.016578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.016469    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-routing-overlay-volume" pod "7757eb1e-51c8-4498-9bed-ada982488167" (UID: "7757eb1e-51c8-4498-9bed-ada982488167"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:09.016702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.016682    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-plugins-config-volume" pod "7757eb1e-51c8-4498-9bed-ada982488167" (UID: "7757eb1e-51c8-4498-9bed-ada982488167"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:09.016780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.016735    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-plugins-config-volume\") pod \"7757eb1e-51c8-4498-9bed-ada982488167\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") "
Oct 30 13:33:09.016847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.016822    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-provider-credentials\") pod \"7757eb1e-51c8-4498-9bed-ada982488167\" (UID: \"7757eb1e-51c8-4498-9bed-ada982488167\") "
Oct 30 13:33:09.017570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.017537    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:09.017570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.017565    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7757eb1e-51c8-4498-9bed-ada982488167-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:09.019160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.019133    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-kube-api-access-f9lbb" pod "7757eb1e-51c8-4498-9bed-ada982488167" (UID: "7757eb1e-51c8-4498-9bed-ada982488167"). InnerVolumeSpecName "kube-api-access-f9lbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:09.019702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.019683    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-provider-credentials" pod "7757eb1e-51c8-4498-9bed-ada982488167" (UID: "7757eb1e-51c8-4498-9bed-ada982488167"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:09.119422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.119321    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9lbb\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-kube-api-access-f9lbb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:09.119422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.119353    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7757eb1e-51c8-4498-9bed-ada982488167-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:09.500321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.500283    4993 generic.go:356] "Generic (PLEG): container finished" podID="7757eb1e-51c8-4498-9bed-ada982488167" containerID="94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49" exitCode=0
Oct 30 13:33:09.500540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.500345    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"
Oct 30 13:33:09.500540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.500375    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd" event={"ID":"7757eb1e-51c8-4498-9bed-ada982488167","Type":"ContainerDied","Data":"94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49"}
Oct 30 13:33:09.500540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.500431    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd" event={"ID":"7757eb1e-51c8-4498-9bed-ada982488167","Type":"ContainerDied","Data":"716fdac4c0585969204180335bd40c5d02265a7d4834bb64b90a44ae3d1d5fbb"}
Oct 30 13:33:09.500540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.500459    4993 scope.go:122] "RemoveContainer" containerID="94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49"
Oct 30 13:33:09.513117 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.513085    4993 scope.go:122] "RemoveContainer" containerID="94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49"
Oct 30 13:33:09.513453 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:33:09.513431    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49\": container with ID starting with 94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49 not found: ID does not exist" containerID="94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49"
Oct 30 13:33:09.513534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.513462    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49"} err="failed to get container status \"94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49\": rpc error: code = NotFound desc = could not find container \"94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49\": container with ID starting with 94fbb017b353189e95f20eb4a2121fab63c01e01a27eb0fac7e619bf6565fa49 not found: ID does not exist"
Oct 30 13:33:09.554612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.554576    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"]
Oct 30 13:33:09.565916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.565880    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-6f858nknd"]
Oct 30 13:33:09.694597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:09.694561    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7757eb1e-51c8-4498-9bed-ada982488167" path="/var/lib/kubelet/pods/7757eb1e-51c8-4498-9bed-ada982488167/volumes"
Oct 30 13:33:15.031736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:15.031696    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"]
Oct 30 13:33:15.043342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:15.043310    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de73496d-zgpgv"]
Oct 30 13:33:15.641583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:15.641508    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"]
Oct 30 13:33:15.731648 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:15.731601    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e3f1fa-135a-4f28-a97a-7f5fff516b6e" path="/var/lib/kubelet/pods/78e3f1fa-135a-4f28-a97a-7f5fff516b6e/volumes"
Oct 30 13:33:17.031075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:17.031029    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"]
Oct 30 13:33:17.042683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:17.042646    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dacada95-8vwmw"]
Oct 30 13:33:17.692684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:17.692650    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3688328f-acfd-4f9a-9a6f-43d3f6f66889" path="/var/lib/kubelet/pods/3688328f-acfd-4f9a-9a6f-43d3f6f66889/volumes"
Oct 30 13:33:20.636157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:20.636101    4993 scope.go:122] "RemoveContainer" containerID="bedd1f9ac081b5cc26e397f36b9d1f3a467ea305721e507f061aa42d9827e084"
Oct 30 13:33:20.642230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:20.642156    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2" podUID="e25ae132-ab62-48e7-86e5-1c4517e51365" containerName="payload-processing" containerID="cri-o://6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2" gracePeriod=25
Oct 30 13:33:20.647320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:20.647299    4993 scope.go:122] "RemoveContainer" containerID="b0dbd338fccbea0bb1fb4481049cdf6e88d5e3a521aff61f397095c9845e02bf"
Oct 30 13:33:20.938558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:20.938529    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:33:21.034469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.034431    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-kube-api-access-5hrjb\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-kube-api-access-5hrjb\") pod \"e25ae132-ab62-48e7-86e5-1c4517e51365\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") "
Oct 30 13:33:21.034688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.034557    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-plugins-config-volume\") pod \"e25ae132-ab62-48e7-86e5-1c4517e51365\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") "
Oct 30 13:33:21.034773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.034711    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-provider-credentials\") pod \"e25ae132-ab62-48e7-86e5-1c4517e51365\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") "
Oct 30 13:33:21.034833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.034796    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-routing-overlay-volume\") pod \"e25ae132-ab62-48e7-86e5-1c4517e51365\" (UID: \"e25ae132-ab62-48e7-86e5-1c4517e51365\") "
Oct 30 13:33:21.035185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.035148    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-routing-overlay-volume" pod "e25ae132-ab62-48e7-86e5-1c4517e51365" (UID: "e25ae132-ab62-48e7-86e5-1c4517e51365"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:21.035185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.035162    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-plugins-config-volume" pod "e25ae132-ab62-48e7-86e5-1c4517e51365" (UID: "e25ae132-ab62-48e7-86e5-1c4517e51365"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:21.035702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.035674    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:21.035863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.035707    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e25ae132-ab62-48e7-86e5-1c4517e51365-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:21.037188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.037157    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-provider-credentials" pod "e25ae132-ab62-48e7-86e5-1c4517e51365" (UID: "e25ae132-ab62-48e7-86e5-1c4517e51365"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:21.037649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.037624    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-kube-api-access-5hrjb" pod "e25ae132-ab62-48e7-86e5-1c4517e51365" (UID: "e25ae132-ab62-48e7-86e5-1c4517e51365"). InnerVolumeSpecName "kube-api-access-5hrjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:21.136887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.136852    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hrjb\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-kube-api-access-5hrjb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:21.136887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.136883    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e25ae132-ab62-48e7-86e5-1c4517e51365-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:21.613548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.613510    4993 generic.go:356] "Generic (PLEG): container finished" podID="e25ae132-ab62-48e7-86e5-1c4517e51365" containerID="6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2" exitCode=0
Oct 30 13:33:21.613755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.613566    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"
Oct 30 13:33:21.613755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.613600    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2" event={"ID":"e25ae132-ab62-48e7-86e5-1c4517e51365","Type":"ContainerDied","Data":"6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2"}
Oct 30 13:33:21.613755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.613667    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2" event={"ID":"e25ae132-ab62-48e7-86e5-1c4517e51365","Type":"ContainerDied","Data":"81e199ccbb7056c6b0cdc5dd4928337d31b74eb3d09afbe37a59c2638d209fd8"}
Oct 30 13:33:21.613755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.613699    4993 scope.go:122] "RemoveContainer" containerID="6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2"
Oct 30 13:33:21.625831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.625809    4993 scope.go:122] "RemoveContainer" containerID="6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2"
Oct 30 13:33:21.626132 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:33:21.626111    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2\": container with ID starting with 6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2 not found: ID does not exist" containerID="6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2"
Oct 30 13:33:21.626198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.626146    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2"} err="failed to get container status \"6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2\": rpc error: code = NotFound desc = could not find container \"6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2\": container with ID starting with 6ea253a9c01073fb9571bd800bf8c18adcf65fd09ccdf1ca23e0ae3952e1ddb2 not found: ID does not exist"
Oct 30 13:33:21.666685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.666641    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"]
Oct 30 13:33:21.680992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.680945    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-c79fj5vq2"]
Oct 30 13:33:21.693783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:21.693753    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25ae132-ab62-48e7-86e5-1c4517e51365" path="/var/lib/kubelet/pods/e25ae132-ab62-48e7-86e5-1c4517e51365/volumes"
Oct 30 13:33:26.042103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:26.042057    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"]
Oct 30 13:33:26.052924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:26.052885    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33b7579-7xlqg"]
Oct 30 13:33:27.692925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:27.692877    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f706e66d-43ad-4c6e-b7dd-18a9031ed813" path="/var/lib/kubelet/pods/f706e66d-43ad-4c6e-b7dd-18a9031ed813/volumes"
Oct 30 13:33:28.186272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.186233    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"]
Oct 30 13:33:28.188107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.188077    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e25ae132-ab62-48e7-86e5-1c4517e51365" containerName="payload-processing"
Oct 30 13:33:28.188309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.188295    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25ae132-ab62-48e7-86e5-1c4517e51365" containerName="payload-processing"
Oct 30 13:33:28.188423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.188413    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e388c5d0-9415-4eae-9b73-ebc02fcb8e13" containerName="revoke-keys"
Oct 30 13:33:28.188502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.188493    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e388c5d0-9415-4eae-9b73-ebc02fcb8e13" containerName="revoke-keys"
Oct 30 13:33:28.188610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.188602    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7757eb1e-51c8-4498-9bed-ada982488167" containerName="payload-processing"
Oct 30 13:33:28.188685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.188674    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7757eb1e-51c8-4498-9bed-ada982488167" containerName="payload-processing"
Oct 30 13:33:28.188932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.188920    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25ae132-ab62-48e7-86e5-1c4517e51365" containerName="payload-processing"
Oct 30 13:33:28.189030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.189021    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e388c5d0-9415-4eae-9b73-ebc02fcb8e13" containerName="revoke-keys"
Oct 30 13:33:28.190790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.190617    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7757eb1e-51c8-4498-9bed-ada982488167" containerName="payload-processing"
Oct 30 13:33:28.195228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.195001    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.210158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.210129    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"]
Oct 30 13:33:28.215040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.215014    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"agc-cred-71a1e2fa7da1\""
Oct 30 13:33:28.215273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.215075    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"payload-processing-external-model-e2e--65402e9a-dockercfg-tqjxr\""
Oct 30 13:33:28.298230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.290665    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0d9115\""
Oct 30 13:33:28.298230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.290874    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"kube-root-ca.crt\""
Oct 30 13:33:28.298230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.291024    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"routing-overlay\""
Oct 30 13:33:28.298230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.291125    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0d9115\"/\"openshift-service-ca.crt\""
Oct 30 13:33:28.337176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.337124    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.337512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.337486    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s76\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-kube-api-access-58s76\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.337715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.337699    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.338010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.337992    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.446061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.445957    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.446331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.446082    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.446331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.446135    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58s76\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-kube-api-access-58s76\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.446331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.446251    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.447041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.447012    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.447868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.447844    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.452006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.451981    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.481032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.480993    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s76\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-kube-api-access-58s76\") pod \"payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") " pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.513864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.513783    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:28.842938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:28.842910    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"]
Oct 30 13:33:28.845965 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:33:28.845918    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d28b08_8676_4f47_ad18_deb023229412.slice/crio-6271283478b63309d3690a36d0393a5e18391f691cfa63919b3e6f347379e0f5 WatchSource:0}: Error finding container 6271283478b63309d3690a36d0393a5e18391f691cfa63919b3e6f347379e0f5: Status 404 returned error can't find the container with id 6271283478b63309d3690a36d0393a5e18391f691cfa63919b3e6f347379e0f5
Oct 30 13:33:29.032264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:29.032223    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"]
Oct 30 13:33:29.037739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:29.037705    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8f3cfdc-nwljc"]
Oct 30 13:33:29.872562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:29.872521    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90341b80-715d-44ea-87ff-e8e2f69b875e" path="/var/lib/kubelet/pods/90341b80-715d-44ea-87ff-e8e2f69b875e/volumes"
Oct 30 13:33:29.904163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:29.904106    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x" event={"ID":"67d28b08-8676-4f47-ad18-deb023229412","Type":"ContainerStarted","Data":"7e545f869ba59789f571d938da7e8e15b623b12fcef58e4f514ad7f1c191e8f6"}
Oct 30 13:33:29.904509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:29.904472    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x" event={"ID":"67d28b08-8676-4f47-ad18-deb023229412","Type":"ContainerStarted","Data":"6271283478b63309d3690a36d0393a5e18391f691cfa63919b3e6f347379e0f5"}
Oct 30 13:33:29.905978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:29.905954    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:30.003270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:29.997582    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x" podStartSLOduration=1.9975564430000001 podStartE2EDuration="1.997556443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:29.986086441 +0000 UTC m=+3674.461806052" watchObservedRunningTime="2026-10-30 13:33:29.997556443 +0000 UTC m=+3674.473276056"
Oct 30 13:33:33.854002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:33.853952    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"]
Oct 30 13:33:33.857242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:33.857180    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:37.310231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.307383    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"]
Oct 30 13:33:37.316242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.315863    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.360340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.341235    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:33:37.360340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.359488    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:33:37.360340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.359553    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:33:37.360340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.359628    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:33:37.469396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.469357    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.469751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.469734    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bq5\" (UniqueName: \"kubernetes.io/projected/f667967a-7a0c-4e2b-8bd0-2e32980755c9-kube-api-access-s5bq5\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.469924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.469913    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.479078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.479033    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"]
Oct 30 13:33:37.584287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.576277    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.584287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.576551    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.584287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.576762    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bq5\" (UniqueName: \"kubernetes.io/projected/f667967a-7a0c-4e2b-8bd0-2e32980755c9-kube-api-access-s5bq5\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.608229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.597103    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.625262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.622923    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.636242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.630010    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bq5\" (UniqueName: \"kubernetes.io/projected/f667967a-7a0c-4e2b-8bd0-2e32980755c9-kube-api-access-s5bq5\") pod \"maas-api-revoke-sub-c966c19d-gbr26\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:37.786456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:37.783459    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:38.037444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:38.037397    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"]
Oct 30 13:33:38.041965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:38.041926    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c6c411-f85tf"]
Oct 30 13:33:38.133160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:38.133129    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"]
Oct 30 13:33:38.135892 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:33:38.135852    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf667967a_7a0c_4e2b_8bd0_2e32980755c9.slice/crio-7f5e6428bf451e7ddc551ea396ab8eb8e646b67021d34a4b9278beee50e129fe WatchSource:0}: Error finding container 7f5e6428bf451e7ddc551ea396ab8eb8e646b67021d34a4b9278beee50e129fe: Status 404 returned error can't find the container with id 7f5e6428bf451e7ddc551ea396ab8eb8e646b67021d34a4b9278beee50e129fe
Oct 30 13:33:38.573525 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:33:38.573485    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:33:38.855146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:38.855033    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x" podUID="67d28b08-8676-4f47-ad18-deb023229412" containerName="payload-processing" containerID="cri-o://7e545f869ba59789f571d938da7e8e15b623b12fcef58e4f514ad7f1c191e8f6" gracePeriod=25
Oct 30 13:33:38.993141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:38.993098    4993 generic.go:356] "Generic (PLEG): container finished" podID="67d28b08-8676-4f47-ad18-deb023229412" containerID="7e545f869ba59789f571d938da7e8e15b623b12fcef58e4f514ad7f1c191e8f6" exitCode=0
Oct 30 13:33:38.993354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:38.993172    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x" event={"ID":"67d28b08-8676-4f47-ad18-deb023229412","Type":"ContainerDied","Data":"7e545f869ba59789f571d938da7e8e15b623b12fcef58e4f514ad7f1c191e8f6"}
Oct 30 13:33:38.995575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:38.995542    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26" event={"ID":"f667967a-7a0c-4e2b-8bd0-2e32980755c9","Type":"ContainerStarted","Data":"b2ec60823bd57d1eab766fcbbc5779f35aba051f1c76537c301e17f1d9acbe69"}
Oct 30 13:33:38.995725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:38.995601    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26" event={"ID":"f667967a-7a0c-4e2b-8bd0-2e32980755c9","Type":"ContainerStarted","Data":"7f5e6428bf451e7ddc551ea396ab8eb8e646b67021d34a4b9278beee50e129fe"}
Oct 30 13:33:39.041875 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.041503    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26" podStartSLOduration=2.041480767 podStartE2EDuration="2.041480767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:39.03509899 +0000 UTC m=+3683.510818601" watchObservedRunningTime="2026-10-30 13:33:39.041480767 +0000 UTC m=+3683.517200379"
Oct 30 13:33:39.159487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.159457    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:39.307591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.307552    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-provider-credentials\") pod \"67d28b08-8676-4f47-ad18-deb023229412\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") "
Oct 30 13:33:39.307782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.307740    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-plugins-config-volume\") pod \"67d28b08-8676-4f47-ad18-deb023229412\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") "
Oct 30 13:33:39.307827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.307779    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-kube-api-access-58s76\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-kube-api-access-58s76\") pod \"67d28b08-8676-4f47-ad18-deb023229412\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") "
Oct 30 13:33:39.307827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.307807    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-routing-overlay-volume\") pod \"67d28b08-8676-4f47-ad18-deb023229412\" (UID: \"67d28b08-8676-4f47-ad18-deb023229412\") "
Oct 30 13:33:39.308178 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.308154    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-routing-overlay-volume" pod "67d28b08-8676-4f47-ad18-deb023229412" (UID: "67d28b08-8676-4f47-ad18-deb023229412"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:39.308313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.308232    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-plugins-config-volume" pod "67d28b08-8676-4f47-ad18-deb023229412" (UID: "67d28b08-8676-4f47-ad18-deb023229412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:39.308443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.308426    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:39.308513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.308444    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d28b08-8676-4f47-ad18-deb023229412-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:39.309879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.309855    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-kube-api-access-58s76" pod "67d28b08-8676-4f47-ad18-deb023229412" (UID: "67d28b08-8676-4f47-ad18-deb023229412"). InnerVolumeSpecName "kube-api-access-58s76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:39.309978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.309893    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-provider-credentials" pod "67d28b08-8676-4f47-ad18-deb023229412" (UID: "67d28b08-8676-4f47-ad18-deb023229412"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:39.410084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.409996    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58s76\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-kube-api-access-58s76\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:39.410084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.410030    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d28b08-8676-4f47-ad18-deb023229412-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:39.694055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:39.693974    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c277ff81-ced3-4e2c-9cb6-896597a5d9bd" path="/var/lib/kubelet/pods/c277ff81-ced3-4e2c-9cb6-896597a5d9bd/volumes"
Oct 30 13:33:40.056272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.056143    4993 generic.go:356] "Generic (PLEG): container finished" podID="f667967a-7a0c-4e2b-8bd0-2e32980755c9" containerID="b2ec60823bd57d1eab766fcbbc5779f35aba051f1c76537c301e17f1d9acbe69" exitCode=0
Oct 30 13:33:40.056708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.056353    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26" event={"ID":"f667967a-7a0c-4e2b-8bd0-2e32980755c9","Type":"ContainerDied","Data":"b2ec60823bd57d1eab766fcbbc5779f35aba051f1c76537c301e17f1d9acbe69"}
Oct 30 13:33:40.083113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.060187    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x" event={"ID":"67d28b08-8676-4f47-ad18-deb023229412","Type":"ContainerDied","Data":"6271283478b63309d3690a36d0393a5e18391f691cfa63919b3e6f347379e0f5"}
Oct 30 13:33:40.083113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.060281    4993 scope.go:122] "RemoveContainer" containerID="7e545f869ba59789f571d938da7e8e15b623b12fcef58e4f514ad7f1c191e8f6"
Oct 30 13:33:40.083113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.060454    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"
Oct 30 13:33:40.145500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.145462    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"]
Oct 30 13:33:40.158334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.158295    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0d9115/payload-processing-external-model-e2e-extmod-b-0d9115-857cdcx5x"]
Oct 30 13:33:40.913725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.913680    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"]
Oct 30 13:33:40.914743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.914651    4993 cpu_manager.go:414] "containerMap: removing container" podUID="67d28b08-8676-4f47-ad18-deb023229412" containerName="payload-processing"
Oct 30 13:33:40.914743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.914675    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d28b08-8676-4f47-ad18-deb023229412" containerName="payload-processing"
Oct 30 13:33:40.914944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.914912    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d28b08-8676-4f47-ad18-deb023229412" containerName="payload-processing"
Oct 30 13:33:40.920331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.920304    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:40.939934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.939419    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9cc0e6-openshift-default-dockercfg-8zqv6\""
Oct 30 13:33:40.979846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:40.979798    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"]
Oct 30 13:33:41.047902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.047866    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-certs\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.048156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.047958    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-envoy\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.048156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.048030    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-socket\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.048156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.048066    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgcdc\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-kube-api-access-mgcdc\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.048156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.048106    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-token\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.048428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.048391    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istiod-ca-cert\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.048533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.048504    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-podinfo\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.048601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.048557    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-credential-socket\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.048640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.048619    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-data\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.152451    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-data\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.152545    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-certs\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.152633    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-envoy\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.152716    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-socket\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.152760    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgcdc\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-kube-api-access-mgcdc\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.152828    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-token\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.152947    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istiod-ca-cert\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.153005    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-podinfo\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.153054    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-credential-socket\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.153588    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-credential-socket\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.153864    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-data\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.161233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.154133    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-certs\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.174292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.172597    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-socket\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.174292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.173557    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istiod-ca-cert\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.189227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.186639    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-podinfo\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.227329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.226762    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-envoy\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.227329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.227303    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-token\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.228312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.228071    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgcdc\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-kube-api-access-mgcdc\") pod \"e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") " pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.250154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.249870    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:41.623126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.623094    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:41.699774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.699682    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d28b08-8676-4f47-ad18-deb023229412" path="/var/lib/kubelet/pods/67d28b08-8676-4f47-ad18-deb023229412/volumes"
Oct 30 13:33:41.810815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.810780    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-xks-ca\") pod \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") "
Oct 30 13:33:41.811002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.810881    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-openshift-ca\") pod \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") "
Oct 30 13:33:41.811002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.810914    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f667967a-7a0c-4e2b-8bd0-2e32980755c9-kube-api-access-s5bq5\" (UniqueName: \"kubernetes.io/projected/f667967a-7a0c-4e2b-8bd0-2e32980755c9-kube-api-access-s5bq5\") pod \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\" (UID: \"f667967a-7a0c-4e2b-8bd0-2e32980755c9\") "
Oct 30 13:33:41.811486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.811339    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-openshift-ca" pod "f667967a-7a0c-4e2b-8bd0-2e32980755c9" (UID: "f667967a-7a0c-4e2b-8bd0-2e32980755c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:41.811766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.811740    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:41.813393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.813366    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-xks-ca" pod "f667967a-7a0c-4e2b-8bd0-2e32980755c9" (UID: "f667967a-7a0c-4e2b-8bd0-2e32980755c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:33:41.813823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.813792    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f667967a-7a0c-4e2b-8bd0-2e32980755c9-kube-api-access-s5bq5" pod "f667967a-7a0c-4e2b-8bd0-2e32980755c9" (UID: "f667967a-7a0c-4e2b-8bd0-2e32980755c9"). InnerVolumeSpecName "kube-api-access-s5bq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:41.922238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.915452    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f667967a-7a0c-4e2b-8bd0-2e32980755c9-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:41.922238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.915499    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bq5\" (UniqueName: \"kubernetes.io/projected/f667967a-7a0c-4e2b-8bd0-2e32980755c9-kube-api-access-s5bq5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:41.945381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.942331    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"]
Oct 30 13:33:41.952881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.952838    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:41.953044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.952935    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:41.953044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:41.952978    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:42.092230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:42.090923    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" event={"ID":"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9","Type":"ContainerStarted","Data":"255cf86de66be1b33cc2ba702c6e5b108f9c02f9be9c4db62b4e7573b760d94f"}
Oct 30 13:33:42.092230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:42.091010    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" event={"ID":"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9","Type":"ContainerStarted","Data":"7a314cb11ef3169ee03d4efef0aafd8bd1e9708bbefa583deb920d04286d1188"}
Oct 30 13:33:42.099236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:42.097432    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26" event={"ID":"f667967a-7a0c-4e2b-8bd0-2e32980755c9","Type":"ContainerDied","Data":"7f5e6428bf451e7ddc551ea396ab8eb8e646b67021d34a4b9278beee50e129fe"}
Oct 30 13:33:42.099236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:42.097489    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f5e6428bf451e7ddc551ea396ab8eb8e646b67021d34a4b9278beee50e129fe"
Oct 30 13:33:42.099236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:42.097551    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"
Oct 30 13:33:42.154894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:42.149965    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podStartSLOduration=2.149946308 podStartE2EDuration="2.149946308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:42.148173683 +0000 UTC m=+3686.623893294" watchObservedRunningTime="2026-10-30 13:33:42.149946308 +0000 UTC m=+3686.625665917"
Oct 30 13:33:42.251223 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:42.251157    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:43.259122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:43.259083    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:44.115574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:44.115540    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:44.116570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:44.116549    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:33:46.033741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:46.033699    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"]
Oct 30 13:33:46.045013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:46.044976    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e101951-pcg9p"]
Oct 30 13:33:47.694105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:47.694064    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510442de-7698-44a3-9c8c-accb239699df" path="/var/lib/kubelet/pods/510442de-7698-44a3-9c8c-accb239699df/volumes"
Oct 30 13:33:48.269008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.268971    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"]
Oct 30 13:33:48.269616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.269597    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f667967a-7a0c-4e2b-8bd0-2e32980755c9" containerName="revoke-keys"
Oct 30 13:33:48.269684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.269632    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f667967a-7a0c-4e2b-8bd0-2e32980755c9" containerName="revoke-keys"
Oct 30 13:33:48.269771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.269760    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f667967a-7a0c-4e2b-8bd0-2e32980755c9" containerName="revoke-keys"
Oct 30 13:33:48.274675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.274656    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.298102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.298066    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:33:48.305651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.305612    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"]
Oct 30 13:33:48.309127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.309095    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:33:48.309311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.309185    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:33:48.310672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.310646    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:33:48.351240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.351184    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.351442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.351387    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.351442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.351427    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgw4\" (UniqueName: \"kubernetes.io/projected/5e0e335c-5081-4dc0-990d-1797c7cbb766-kube-api-access-4fgw4\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.452828 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.452782    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.453035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.452947    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.453035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.452976    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgw4\" (UniqueName: \"kubernetes.io/projected/5e0e335c-5081-4dc0-990d-1797c7cbb766-kube-api-access-4fgw4\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.453872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.453842    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.455372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.455351    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.478782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.478751    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgw4\" (UniqueName: \"kubernetes.io/projected/5e0e335c-5081-4dc0-990d-1797c7cbb766-kube-api-access-4fgw4\") pod \"maas-api-revoke-sub-33f1fe5a-6w4vd\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.607751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.607663    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:48.918224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:48.918180    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"]
Oct 30 13:33:48.919774 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:33:48.919741    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0e335c_5081_4dc0_990d_1797c7cbb766.slice/crio-12be86b6b006b2748a4b37b285be380d41c9b6f8705ddd1001ee16a6c38a0611 WatchSource:0}: Error finding container 12be86b6b006b2748a4b37b285be380d41c9b6f8705ddd1001ee16a6c38a0611: Status 404 returned error can't find the container with id 12be86b6b006b2748a4b37b285be380d41c9b6f8705ddd1001ee16a6c38a0611
Oct 30 13:33:49.161017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:49.160982    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd" event={"ID":"5e0e335c-5081-4dc0-990d-1797c7cbb766","Type":"ContainerStarted","Data":"12be86b6b006b2748a4b37b285be380d41c9b6f8705ddd1001ee16a6c38a0611"}
Oct 30 13:33:50.030948 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:50.030909    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"]
Oct 30 13:33:50.042578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:50.042535    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f48b40c-wmk7l"]
Oct 30 13:33:50.172749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:50.172699    4993 generic.go:356] "Generic (PLEG): container finished" podID="5e0e335c-5081-4dc0-990d-1797c7cbb766" containerID="d54736adb4ba0ce5136b7ccc1624a935651ed9750eb472c8e450bbb5d7ba4887" exitCode=0
Oct 30 13:33:50.172960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:50.172772    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd" event={"ID":"5e0e335c-5081-4dc0-990d-1797c7cbb766","Type":"ContainerDied","Data":"d54736adb4ba0ce5136b7ccc1624a935651ed9750eb472c8e450bbb5d7ba4887"}
Oct 30 13:33:51.368157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.368121    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:51.504177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.504134    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0e335c-5081-4dc0-990d-1797c7cbb766-kube-api-access-4fgw4\" (UniqueName: \"kubernetes.io/projected/5e0e335c-5081-4dc0-990d-1797c7cbb766-kube-api-access-4fgw4\") pod \"5e0e335c-5081-4dc0-990d-1797c7cbb766\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") "
Oct 30 13:33:51.504381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.504236    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-openshift-ca\") pod \"5e0e335c-5081-4dc0-990d-1797c7cbb766\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") "
Oct 30 13:33:51.504475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.504461    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-xks-ca\") pod \"5e0e335c-5081-4dc0-990d-1797c7cbb766\" (UID: \"5e0e335c-5081-4dc0-990d-1797c7cbb766\") "
Oct 30 13:33:51.504710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.504682    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-openshift-ca" pod "5e0e335c-5081-4dc0-990d-1797c7cbb766" (UID: "5e0e335c-5081-4dc0-990d-1797c7cbb766"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:51.505031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.505004    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:51.507072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.507036    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-xks-ca" pod "5e0e335c-5081-4dc0-990d-1797c7cbb766" (UID: "5e0e335c-5081-4dc0-990d-1797c7cbb766"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:33:51.507072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.507053    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0e335c-5081-4dc0-990d-1797c7cbb766-kube-api-access-4fgw4" pod "5e0e335c-5081-4dc0-990d-1797c7cbb766" (UID: "5e0e335c-5081-4dc0-990d-1797c7cbb766"). InnerVolumeSpecName "kube-api-access-4fgw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:51.606695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.606651    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0e335c-5081-4dc0-990d-1797c7cbb766-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:51.606695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.606691    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgw4\" (UniqueName: \"kubernetes.io/projected/5e0e335c-5081-4dc0-990d-1797c7cbb766-kube-api-access-4fgw4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:33:51.697174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:51.697126    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93eeb04b-6c30-467c-a721-ceb32f9c086e" path="/var/lib/kubelet/pods/93eeb04b-6c30-467c-a721-ceb32f9c086e/volumes"
Oct 30 13:33:52.193094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:52.193058    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"
Oct 30 13:33:52.193313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:52.193087    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd" event={"ID":"5e0e335c-5081-4dc0-990d-1797c7cbb766","Type":"ContainerDied","Data":"12be86b6b006b2748a4b37b285be380d41c9b6f8705ddd1001ee16a6c38a0611"}
Oct 30 13:33:52.193313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:33:52.193160    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12be86b6b006b2748a4b37b285be380d41c9b6f8705ddd1001ee16a6c38a0611"
Oct 30 13:34:01.229631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:01.229583    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"]
Oct 30 13:34:02.036118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:02.036071    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"]
Oct 30 13:34:02.040478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:02.040435    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18996bd-xrsvd"]
Oct 30 13:34:03.030797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:03.030763    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"]
Oct 30 13:34:03.042193 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:03.042157    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8721350-8v5mr"]
Oct 30 13:34:03.696116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:03.696076    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f40961f-88bc-45c8-8ff0-876a52e1353d" path="/var/lib/kubelet/pods/6f40961f-88bc-45c8-8ff0-876a52e1353d/volumes"
Oct 30 13:34:03.696741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:03.696717    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee72959e-de1a-4721-a621-8ccdad306a01" path="/var/lib/kubelet/pods/ee72959e-de1a-4721-a621-8ccdad306a01/volumes"
Oct 30 13:34:05.498267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:05.497846    4993 generic.go:356] "Generic (PLEG): container finished" podID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerID="6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b" exitCode=0
Oct 30 13:34:05.498267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:05.498127    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" event={"ID":"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a","Type":"ContainerDied","Data":"6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b"}
Oct 30 13:34:05.506243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:05.499405    4993 scope.go:122] "RemoveContainer" containerID="6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b"
Oct 30 13:34:05.514798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:05.514464    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:05.514798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:05.514575    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:05.514798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:05.514628    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:06.230924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.230882    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n" podUID="3647e0ae-cf37-400e-b943-ceed09f879cd" containerName="payload-processing" containerID="cri-o://b4c58072e0691b9545a5a9fd8e7d3ff4a8b636558e1caeb49c7383af5c9b8728" gracePeriod=25
Oct 30 13:34:06.523597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.523504    4993 generic.go:356] "Generic (PLEG): container finished" podID="3647e0ae-cf37-400e-b943-ceed09f879cd" containerID="b4c58072e0691b9545a5a9fd8e7d3ff4a8b636558e1caeb49c7383af5c9b8728" exitCode=0
Oct 30 13:34:06.523980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.523684    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n" event={"ID":"3647e0ae-cf37-400e-b943-ceed09f879cd","Type":"ContainerDied","Data":"b4c58072e0691b9545a5a9fd8e7d3ff4a8b636558e1caeb49c7383af5c9b8728"}
Oct 30 13:34:06.526426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.526386    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" event={"ID":"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a","Type":"ContainerStarted","Data":"1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68"}
Oct 30 13:34:06.601784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.601746    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:34:06.714322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.714282    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-provider-credentials\") pod \"3647e0ae-cf37-400e-b943-ceed09f879cd\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") "
Oct 30 13:34:06.714568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.714550    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-routing-overlay-volume\") pod \"3647e0ae-cf37-400e-b943-ceed09f879cd\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") "
Oct 30 13:34:06.714647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.714594    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-kube-api-access-p92v6\") pod \"3647e0ae-cf37-400e-b943-ceed09f879cd\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") "
Oct 30 13:34:06.714706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.714650    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-plugins-config-volume\") pod \"3647e0ae-cf37-400e-b943-ceed09f879cd\" (UID: \"3647e0ae-cf37-400e-b943-ceed09f879cd\") "
Oct 30 13:34:06.715002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.714978    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-routing-overlay-volume" pod "3647e0ae-cf37-400e-b943-ceed09f879cd" (UID: "3647e0ae-cf37-400e-b943-ceed09f879cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:06.715137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.715046    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-plugins-config-volume" pod "3647e0ae-cf37-400e-b943-ceed09f879cd" (UID: "3647e0ae-cf37-400e-b943-ceed09f879cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:06.715302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.715285    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.715350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.715304    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647e0ae-cf37-400e-b943-ceed09f879cd-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.716501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.716475    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-provider-credentials" pod "3647e0ae-cf37-400e-b943-ceed09f879cd" (UID: "3647e0ae-cf37-400e-b943-ceed09f879cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:06.716668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.716645    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-kube-api-access-p92v6" pod "3647e0ae-cf37-400e-b943-ceed09f879cd" (UID: "3647e0ae-cf37-400e-b943-ceed09f879cd"). InnerVolumeSpecName "kube-api-access-p92v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:06.817964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.817864    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-kube-api-access-p92v6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.817964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:06.817910    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3647e0ae-cf37-400e-b943-ceed09f879cd-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:07.297230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.296404    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:34:07.305229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.301389    4993 patch_prober.go:29] interesting pod/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 30 13:34:07.305229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.301471    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 30 13:34:07.537264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.537218    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n" event={"ID":"3647e0ae-cf37-400e-b943-ceed09f879cd","Type":"ContainerDied","Data":"011fe78433ddf3687bbfe62426eb5554363869d938f141469648882035defcb4"}
Oct 30 13:34:07.537264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.537245    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"
Oct 30 13:34:07.537797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.537296    4993 scope.go:122] "RemoveContainer" containerID="b4c58072e0691b9545a5a9fd8e7d3ff4a8b636558e1caeb49c7383af5c9b8728"
Oct 30 13:34:07.629384 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.629337    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"]
Oct 30 13:34:07.632134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.632104    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-699fnmm2n"]
Oct 30 13:34:07.697182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:07.697152    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3647e0ae-cf37-400e-b943-ceed09f879cd" path="/var/lib/kubelet/pods/3647e0ae-cf37-400e-b943-ceed09f879cd/volumes"
Oct 30 13:34:08.287106 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.287058    4993 patch_prober.go:29] interesting pod/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 30 13:34:08.287342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.287160    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 30 13:34:08.487231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.484563    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"]
Oct 30 13:34:08.504287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.492626    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3647e0ae-cf37-400e-b943-ceed09f879cd" containerName="payload-processing"
Oct 30 13:34:08.504287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.492660    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3647e0ae-cf37-400e-b943-ceed09f879cd" containerName="payload-processing"
Oct 30 13:34:08.504287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.492686    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5e0e335c-5081-4dc0-990d-1797c7cbb766" containerName="revoke-keys"
Oct 30 13:34:08.504287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.492695    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0e335c-5081-4dc0-990d-1797c7cbb766" containerName="revoke-keys"
Oct 30 13:34:08.504287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.492923    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3647e0ae-cf37-400e-b943-ceed09f879cd" containerName="payload-processing"
Oct 30 13:34:08.504287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.492942    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0e335c-5081-4dc0-990d-1797c7cbb766" containerName="revoke-keys"
Oct 30 13:34:08.519555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.518550    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.520166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.520132    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"]
Oct 30 13:34:08.668312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.666659    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-22efcf-openshift-default-dockercfg-rh6lg\""
Oct 30 13:34:08.751679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.740446    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr6w6\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-kube-api-access-sr6w6\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.751679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.740498    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00cfe34-6ca8-4a28-a921-968f1368fb10-istiod-ca-cert\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.751679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.740534    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-envoy\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.751679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.740560    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-data\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.751679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.740620    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.784235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.752487    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-token\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.784235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.752567    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-certs\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.784235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.752596    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-podinfo\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.784235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.752682    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-credential-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.798286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.788316    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.950679    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-token\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.950818    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-certs\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.950858    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-podinfo\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.950937    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-credential-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.951382    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr6w6\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-kube-api-access-sr6w6\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.951429    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00cfe34-6ca8-4a28-a921-968f1368fb10-istiod-ca-cert\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.951459    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-envoy\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.951485    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-data\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.951559    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:08.993250 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:34:08.985137    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:34:08.995086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:08.995053    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.038868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.038835    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-credential-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.039176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.039139    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-certs\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.078330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.065948    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00cfe34-6ca8-4a28-a921-968f1368fb10-istiod-ca-cert\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.078330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.077959    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-data\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.134244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.126873    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"]
Oct 30 13:34:09.134244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.131953    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.238003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.237966    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-podinfo\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.245234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.239892    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ba646b-openshift-default-dockercfg-wjbl8\""
Oct 30 13:34:09.245471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.241352    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-envoy\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.245689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.245670    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-token\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.245826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.245808    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istiod-ca-cert\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.245971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.245956    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-data\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.246185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.246169    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.246398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.246380    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-credential-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.246531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.246517    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-certs\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.246668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.246651    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-podinfo\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.246772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.246757    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9jj\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-kube-api-access-4d9jj\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.253723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.253696    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-token\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.265908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.265875    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-envoy\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.268705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.268673    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"]
Oct 30 13:34:09.291768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.291268    4993 patch_prober.go:29] interesting pod/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 30 13:34:09.291768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.291357    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 30 13:34:09.313232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.310418    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr6w6\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-kube-api-access-sr6w6\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.397293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.349751    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-envoy\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.405385    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-token\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.405443    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istiod-ca-cert\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.405510    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-data\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.405640    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.405692    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-credential-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.405731    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-certs\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.405781    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-podinfo\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.405809    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9jj\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-kube-api-access-4d9jj\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.407236    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istiod-ca-cert\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.407519    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-data\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.407760    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.407986    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-credential-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.414293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.408238    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-certs\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.456640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.456601    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:09.464239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.458128    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-podinfo\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.726258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.724935    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-token\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.726258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.725201    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-envoy\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.726258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.726057    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9jj\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-kube-api-access-4d9jj\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:09.730772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:09.727803    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:10.087519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.087473    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"]
Oct 30 13:34:10.098136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.098099    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:10.099407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.098372    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:10.099407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.098415    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:10.143628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.143589    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"]
Oct 30 13:34:10.147951 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:34:10.147921    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a44edeb_97be_4ef5_878c_7de3abd1bedd.slice/crio-9e628b314196ec40a04a5c70d6a86031c746999ed674f57aca216f85ccf42150 WatchSource:0}: Error finding container 9e628b314196ec40a04a5c70d6a86031c746999ed674f57aca216f85ccf42150: Status 404 returned error can't find the container with id 9e628b314196ec40a04a5c70d6a86031c746999ed674f57aca216f85ccf42150
Oct 30 13:34:10.151133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.151029    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:10.151396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.151294    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:10.151396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.151361    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:10.289244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.286644    4993 patch_prober.go:29] interesting pod/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 30 13:34:10.289244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.286706    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 30 13:34:10.738372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.737790    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" event={"ID":"d00cfe34-6ca8-4a28-a921-968f1368fb10","Type":"ContainerStarted","Data":"445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c"}
Oct 30 13:34:10.738372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.737866    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" event={"ID":"d00cfe34-6ca8-4a28-a921-968f1368fb10","Type":"ContainerStarted","Data":"6a1fc055151016069cc59f258ff5b4b91fbab64f14e16b83589eee0e84180b08"}
Oct 30 13:34:10.741566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.741527    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" event={"ID":"4a44edeb-97be-4ef5-878c-7de3abd1bedd","Type":"ContainerStarted","Data":"a8ceb1b41a76b585a16af3c45d43fc90dda8f5fdfbcad8287f25f21021694c0c"}
Oct 30 13:34:10.741744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.741593    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" event={"ID":"4a44edeb-97be-4ef5-878c-7de3abd1bedd","Type":"ContainerStarted","Data":"9e628b314196ec40a04a5c70d6a86031c746999ed674f57aca216f85ccf42150"}
Oct 30 13:34:10.822766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.822655    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podStartSLOduration=2.822634048 podStartE2EDuration="2.822634048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:10.820982772 +0000 UTC m=+3715.296702393" watchObservedRunningTime="2026-10-30 13:34:10.822634048 +0000 UTC m=+3715.298353659"
Oct 30 13:34:10.912571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:10.912468    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" podStartSLOduration=1.912442875 podStartE2EDuration="1.912442875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:10.893451756 +0000 UTC m=+3715.369171380" watchObservedRunningTime="2026-10-30 13:34:10.912442875 +0000 UTC m=+3715.388162488"
Oct 30 13:34:11.291229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.289201    4993 patch_prober.go:29] interesting pod/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 30 13:34:11.291229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.289288    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 30 13:34:11.464230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.463386    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:11.484233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.481233    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:11.728952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.728914    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:11.736096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.736062    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:11.751939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.751911    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:11.764335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.764301    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:11.764630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.764613    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:34:11.764788 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:11.764775    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:34:12.288234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:12.287437    4993 patch_prober.go:29] interesting pod/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Oct 30 13:34:12.288234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:12.287501    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Oct 30 13:34:13.292601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:13.292567    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:34:13.293953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:13.293930    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:34:15.055077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:15.055036    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"]
Oct 30 13:34:15.066309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:15.066274    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101d25bb-64457"]
Oct 30 13:34:16.029279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:16.029242    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0888d296-9a88-4d16-860b-56bac84c654a" path="/var/lib/kubelet/pods/0888d296-9a88-4d16-860b-56bac84c654a/volumes"
Oct 30 13:34:20.784055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:20.784014    4993 scope.go:122] "RemoveContainer" containerID="fc35b64ddf36cc0c247cbd15f760a5e5d9293ea66a20a2187f68fc15c730122c"
Oct 30 13:34:20.797906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:20.797875    4993 scope.go:122] "RemoveContainer" containerID="9aa5a69b52fb1ebfd9b809a2f5f40c1bed813d89a08f1c44651a94a7f39c83ba"
Oct 30 13:34:20.809609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:20.809572    4993 scope.go:122] "RemoveContainer" containerID="4f82845ef0abd034c0cf6730a733d5f8990a181561b6d3dafac65cf10e89b481"
Oct 30 13:34:20.820632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:20.820609    4993 scope.go:122] "RemoveContainer" containerID="3241458d78c963a763b650d63ab03937cda0ae17ee03081e2859f4198e0ea092"
Oct 30 13:34:29.739774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:29.737119    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:34:29.815334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:29.814252    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:34:32.856685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.856645    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"]
Oct 30 13:34:32.862123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.862093    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:32.887784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.887750    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"]
Oct 30 13:34:32.889341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.889309    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:34:32.889341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.889309    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:34:32.889634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.889612    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:34:32.889731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.889711    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:34:32.972266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.972185    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zth7j\" (UniqueName: \"kubernetes.io/projected/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-kube-api-access-zth7j\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:32.972566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.972538    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:32.972696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:32.972653    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:33.073772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:33.073735    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zth7j\" (UniqueName: \"kubernetes.io/projected/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-kube-api-access-zth7j\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:33.073989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:33.073821    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:33.073989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:33.073873    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:33.074697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:33.074664    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:33.076370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:33.076346    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:33.097395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:33.097362    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zth7j\" (UniqueName: \"kubernetes.io/projected/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-kube-api-access-zth7j\") pod \"maas-api-revoke-sub-f01bdc7d-6qh2m\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:33.182080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:33.181990    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:33.525514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:33.524977    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"]
Oct 30 13:34:33.528817 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:34:33.528783    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b9fff7_ddc2_44f7_bf2d_f0bb489d2a52.slice/crio-953cfc50ed6676b6a5f9c306b969730638e0d47df6acd42a4c304c869337a365 WatchSource:0}: Error finding container 953cfc50ed6676b6a5f9c306b969730638e0d47df6acd42a4c304c869337a365: Status 404 returned error can't find the container with id 953cfc50ed6676b6a5f9c306b969730638e0d47df6acd42a4c304c869337a365
Oct 30 13:34:34.288148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:34.288042    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m" event={"ID":"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52","Type":"ContainerStarted","Data":"8c6652d7d26a2a59b5676e5604ea845a22f495a769c760ec3b05b0ab46a9e075"}
Oct 30 13:34:34.288699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:34.288175    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m" event={"ID":"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52","Type":"ContainerStarted","Data":"953cfc50ed6676b6a5f9c306b969730638e0d47df6acd42a4c304c869337a365"}
Oct 30 13:34:34.336475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:34.334405    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m" podStartSLOduration=2.334390955 podStartE2EDuration="2.334390955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:34.33201965 +0000 UTC m=+3738.807739260" watchObservedRunningTime="2026-10-30 13:34:34.334390955 +0000 UTC m=+3738.810110564"
Oct 30 13:34:35.298088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:35.298046    4993 generic.go:356] "Generic (PLEG): container finished" podID="d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52" containerID="8c6652d7d26a2a59b5676e5604ea845a22f495a769c760ec3b05b0ab46a9e075" exitCode=0
Oct 30 13:34:35.298679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:35.298135    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m" event={"ID":"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52","Type":"ContainerDied","Data":"8c6652d7d26a2a59b5676e5604ea845a22f495a769c760ec3b05b0ab46a9e075"}
Oct 30 13:34:36.487537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.487511    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:36.555488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.555455    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-kube-api-access-zth7j\" (UniqueName: \"kubernetes.io/projected/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-kube-api-access-zth7j\") pod \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") "
Oct 30 13:34:36.555699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.555678    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-openshift-ca\") pod \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") "
Oct 30 13:34:36.555774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.555744    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-xks-ca\") pod \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\" (UID: \"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52\") "
Oct 30 13:34:36.556113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.556084    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-openshift-ca" pod "d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52" (UID: "d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:36.556621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.556589    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:36.557898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.557840    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-kube-api-access-zth7j" pod "d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52" (UID: "d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52"). InnerVolumeSpecName "kube-api-access-zth7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:36.558405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.558381    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-xks-ca" pod "d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52" (UID: "d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:36.657893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.657852    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zth7j\" (UniqueName: \"kubernetes.io/projected/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-kube-api-access-zth7j\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:36.657893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:36.657886    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:37.317157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:37.317129    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"
Oct 30 13:34:37.317329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:37.317161    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m" event={"ID":"d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52","Type":"ContainerDied","Data":"953cfc50ed6676b6a5f9c306b969730638e0d47df6acd42a4c304c869337a365"}
Oct 30 13:34:37.317329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:37.317227    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="953cfc50ed6676b6a5f9c306b969730638e0d47df6acd42a4c304c869337a365"
Oct 30 13:34:38.569771 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:34:38.569728    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:34:40.038723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:40.038682    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"]
Oct 30 13:34:40.049620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:40.049586    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3721a947-dxdqz"]
Oct 30 13:34:41.040774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:41.040733    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"]
Oct 30 13:34:41.052871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:41.052836    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d94d4e-pv42t"]
Oct 30 13:34:41.695873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:41.695830    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1413ba77-60ff-4ce2-978a-250a635b7817" path="/var/lib/kubelet/pods/1413ba77-60ff-4ce2-978a-250a635b7817/volumes"
Oct 30 13:34:41.696564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:41.696544    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ceaba3-f33a-40b2-a76d-abafba657a14" path="/var/lib/kubelet/pods/20ceaba3-f33a-40b2-a76d-abafba657a14/volumes"
Oct 30 13:34:43.775338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.775300    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"]
Oct 30 13:34:43.776110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.776084    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52" containerName="revoke-keys"
Oct 30 13:34:43.776235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.776111    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52" containerName="revoke-keys"
Oct 30 13:34:43.776346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.776332    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52" containerName="revoke-keys"
Oct 30 13:34:43.787844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.787819    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:43.802155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.802127    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"]
Oct 30 13:34:43.804113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.804093    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:34:43.808088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.808051    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:34:43.808202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.808109    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:34:43.808202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.808109    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:34:43.915076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.915043    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:43.915355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.915335    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:43.915412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:43.915375    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtjd\" (UniqueName: \"kubernetes.io/projected/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-kube-api-access-8xtjd\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:44.016980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:44.016947    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtjd\" (UniqueName: \"kubernetes.io/projected/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-kube-api-access-8xtjd\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:44.017162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:44.017058    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:44.017309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:44.017287    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:44.017973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:44.017950    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:44.019500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:44.019476    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:44.040039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:44.039979    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtjd\" (UniqueName: \"kubernetes.io/projected/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-kube-api-access-8xtjd\") pod \"maas-api-revoke-sub-58918eab-wb6tq\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:44.102043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:44.102000    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:44.410106 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:44.410078    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"]
Oct 30 13:34:44.413371 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:34:44.413341    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a086da_85a9_4331_9cbf_f46ab6f3fe86.slice/crio-10acd89ba72d3e1853f3c949f0efc0688c3529af29d2da3b904dce81854de088 WatchSource:0}: Error finding container 10acd89ba72d3e1853f3c949f0efc0688c3529af29d2da3b904dce81854de088: Status 404 returned error can't find the container with id 10acd89ba72d3e1853f3c949f0efc0688c3529af29d2da3b904dce81854de088
Oct 30 13:34:45.424251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:45.424181    4993 generic.go:356] "Generic (PLEG): container finished" podID="d5a086da-85a9-4331-9cbf-f46ab6f3fe86" containerID="18a4016a6f4e6b527c8807111c6a9926df2a99908e0542f71624b27bb20bb5b1" exitCode=0
Oct 30 13:34:45.424670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:45.424272    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq" event={"ID":"d5a086da-85a9-4331-9cbf-f46ab6f3fe86","Type":"ContainerDied","Data":"18a4016a6f4e6b527c8807111c6a9926df2a99908e0542f71624b27bb20bb5b1"}
Oct 30 13:34:45.424670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:45.424334    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq" event={"ID":"d5a086da-85a9-4331-9cbf-f46ab6f3fe86","Type":"ContainerStarted","Data":"10acd89ba72d3e1853f3c949f0efc0688c3529af29d2da3b904dce81854de088"}
Oct 30 13:34:46.623696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.623664    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:46.756325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.756286    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-openshift-ca\") pod \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") "
Oct 30 13:34:46.756567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.756549    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-kube-api-access-8xtjd\" (UniqueName: \"kubernetes.io/projected/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-kube-api-access-8xtjd\") pod \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") "
Oct 30 13:34:46.756638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.756605    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-openshift-ca" pod "d5a086da-85a9-4331-9cbf-f46ab6f3fe86" (UID: "d5a086da-85a9-4331-9cbf-f46ab6f3fe86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:46.756738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.756721    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-xks-ca\") pod \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\" (UID: \"d5a086da-85a9-4331-9cbf-f46ab6f3fe86\") "
Oct 30 13:34:46.757579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.757555    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:46.758859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.758833    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-xks-ca" pod "d5a086da-85a9-4331-9cbf-f46ab6f3fe86" (UID: "d5a086da-85a9-4331-9cbf-f46ab6f3fe86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:46.758962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.758882    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-kube-api-access-8xtjd" pod "d5a086da-85a9-4331-9cbf-f46ab6f3fe86" (UID: "d5a086da-85a9-4331-9cbf-f46ab6f3fe86"). InnerVolumeSpecName "kube-api-access-8xtjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:46.860092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.860000    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xtjd\" (UniqueName: \"kubernetes.io/projected/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-kube-api-access-8xtjd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:46.860381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:46.860357    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a086da-85a9-4331-9cbf-f46ab6f3fe86-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:34:47.451688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:47.451656    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"
Oct 30 13:34:47.451872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:47.451683    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq" event={"ID":"d5a086da-85a9-4331-9cbf-f46ab6f3fe86","Type":"ContainerDied","Data":"10acd89ba72d3e1853f3c949f0efc0688c3529af29d2da3b904dce81854de088"}
Oct 30 13:34:47.451872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:47.451734    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10acd89ba72d3e1853f3c949f0efc0688c3529af29d2da3b904dce81854de088"
Oct 30 13:34:53.069868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:53.069819    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"]
Oct 30 13:34:53.082463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:53.082423    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc055a43-lrxxh"]
Oct 30 13:34:53.696425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:53.696383    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78cf351-9e8a-48ac-b0a0-2fdf44ab3363" path="/var/lib/kubelet/pods/b78cf351-9e8a-48ac-b0a0-2fdf44ab3363/volumes"
Oct 30 13:34:56.240413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.240378    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"]
Oct 30 13:34:56.241965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.241934    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d5a086da-85a9-4331-9cbf-f46ab6f3fe86" containerName="revoke-keys"
Oct 30 13:34:56.242151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.242135    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a086da-85a9-4331-9cbf-f46ab6f3fe86" containerName="revoke-keys"
Oct 30 13:34:56.242517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.242495    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a086da-85a9-4331-9cbf-f46ab6f3fe86" containerName="revoke-keys"
Oct 30 13:34:56.259707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.259675    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.277247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.277032    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:34:56.292088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.291854    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:34:56.292088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.291955    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:34:56.292088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.292044    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:34:56.324230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.313810    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.324230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.313888    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9725\" (UniqueName: \"kubernetes.io/projected/781323a9-1d50-4762-8026-4c2268b7327b-kube-api-access-g9725\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.324230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.314097    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.400566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.400469    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"]
Oct 30 13:34:56.422541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.421512    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.422541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.421591    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9725\" (UniqueName: \"kubernetes.io/projected/781323a9-1d50-4762-8026-4c2268b7327b-kube-api-access-g9725\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.422541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.421799    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.465690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.465656    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.631872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.626570    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9725\" (UniqueName: \"kubernetes.io/projected/781323a9-1d50-4762-8026-4c2268b7327b-kube-api-access-g9725\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:56.824355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:56.824324    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dad88a8-st7cf\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:57.056494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:57.056453    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:34:57.431058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:57.431027    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"]
Oct 30 13:34:57.432540 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:34:57.432512    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781323a9_1d50_4762_8026_4c2268b7327b.slice/crio-007e51ba740914f582f96a9520b103750ad513b6e1d7015e1cf036715c46a6f0 WatchSource:0}: Error finding container 007e51ba740914f582f96a9520b103750ad513b6e1d7015e1cf036715c46a6f0: Status 404 returned error can't find the container with id 007e51ba740914f582f96a9520b103750ad513b6e1d7015e1cf036715c46a6f0
Oct 30 13:34:57.935238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:57.931474    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf" event={"ID":"781323a9-1d50-4762-8026-4c2268b7327b","Type":"ContainerStarted","Data":"007e51ba740914f582f96a9520b103750ad513b6e1d7015e1cf036715c46a6f0"}
Oct 30 13:34:58.943398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:58.943306    4993 generic.go:356] "Generic (PLEG): container finished" podID="781323a9-1d50-4762-8026-4c2268b7327b" containerID="a27aa6a99a6d61f27ab037b3f7cf798ca5ddd9ce996c54ce6bb3772a20802b40" exitCode=0
Oct 30 13:34:58.943860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:34:58.943386    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf" event={"ID":"781323a9-1d50-4762-8026-4c2268b7327b","Type":"ContainerDied","Data":"a27aa6a99a6d61f27ab037b3f7cf798ca5ddd9ce996c54ce6bb3772a20802b40"}
Oct 30 13:35:00.149922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.149893    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:35:00.157396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.157354    4993 patch_prober.go:29] interesting pod/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:35:00.157546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.157412    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:00.248118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.248068    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-openshift-ca\") pod \"781323a9-1d50-4762-8026-4c2268b7327b\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") "
Oct 30 13:35:00.248359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.248317    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781323a9-1d50-4762-8026-4c2268b7327b-kube-api-access-g9725\" (UniqueName: \"kubernetes.io/projected/781323a9-1d50-4762-8026-4c2268b7327b-kube-api-access-g9725\") pod \"781323a9-1d50-4762-8026-4c2268b7327b\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") "
Oct 30 13:35:00.248490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.248456    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-xks-ca\") pod \"781323a9-1d50-4762-8026-4c2268b7327b\" (UID: \"781323a9-1d50-4762-8026-4c2268b7327b\") "
Oct 30 13:35:00.248651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.248544    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-openshift-ca" pod "781323a9-1d50-4762-8026-4c2268b7327b" (UID: "781323a9-1d50-4762-8026-4c2268b7327b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:00.250503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.249218    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:00.251811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.251773    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-xks-ca" pod "781323a9-1d50-4762-8026-4c2268b7327b" (UID: "781323a9-1d50-4762-8026-4c2268b7327b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:00.274269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.263609    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781323a9-1d50-4762-8026-4c2268b7327b-kube-api-access-g9725" pod "781323a9-1d50-4762-8026-4c2268b7327b" (UID: "781323a9-1d50-4762-8026-4c2268b7327b"). InnerVolumeSpecName "kube-api-access-g9725". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:00.350366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.350327    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9725\" (UniqueName: \"kubernetes.io/projected/781323a9-1d50-4762-8026-4c2268b7327b-kube-api-access-g9725\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:00.350366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.350359    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781323a9-1d50-4762-8026-4c2268b7327b-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:00.616122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.614500    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"]
Oct 30 13:35:00.616122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.615806    4993 cpu_manager.go:414] "containerMap: removing container" podUID="781323a9-1d50-4762-8026-4c2268b7327b" containerName="revoke-keys"
Oct 30 13:35:00.616122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.615829    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781323a9-1d50-4762-8026-4c2268b7327b" containerName="revoke-keys"
Oct 30 13:35:00.616714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.616690    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="781323a9-1d50-4762-8026-4c2268b7327b" containerName="revoke-keys"
Oct 30 13:35:00.641077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.639644    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"]
Oct 30 13:35:00.641077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.639777    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.759831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.759788    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8h6f\" (UniqueName: \"kubernetes.io/projected/e3089258-552e-40b5-82a0-d4b7a96e5b58-kube-api-access-j8h6f\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.760033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.759917    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.760033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.759979    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.862338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.862294    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8h6f\" (UniqueName: \"kubernetes.io/projected/e3089258-552e-40b5-82a0-d4b7a96e5b58-kube-api-access-j8h6f\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.862558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.862367    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.862558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.862412    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.863242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.863192    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.865075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.865047    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.890748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.890673    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8h6f\" (UniqueName: \"kubernetes.io/projected/e3089258-552e-40b5-82a0-d4b7a96e5b58-kube-api-access-j8h6f\") pod \"maas-api-revoke-sub-0b68733d-wfs72\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:00.965062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.965028    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"
Oct 30 13:35:00.965289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.965068    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf" event={"ID":"781323a9-1d50-4762-8026-4c2268b7327b","Type":"ContainerDied","Data":"007e51ba740914f582f96a9520b103750ad513b6e1d7015e1cf036715c46a6f0"}
Oct 30 13:35:00.965289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.965131    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="007e51ba740914f582f96a9520b103750ad513b6e1d7015e1cf036715c46a6f0"
Oct 30 13:35:00.982593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:00.982565    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:01.382829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:01.382800    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"]
Oct 30 13:35:01.977218 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:01.977169    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72" event={"ID":"e3089258-552e-40b5-82a0-d4b7a96e5b58","Type":"ContainerStarted","Data":"648a7e47261ab2d24b213aa443dd4024a71c36333e9aab694518b40e5e94da5c"}
Oct 30 13:35:01.977386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:01.977287    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72" event={"ID":"e3089258-552e-40b5-82a0-d4b7a96e5b58","Type":"ContainerStarted","Data":"32c3bc3fa3264b7e7a9e64813c315a205797c3915e2026a53cd6b31738591bb4"}
Oct 30 13:35:02.030868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:02.030810    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72" podStartSLOduration=2.030796423 podStartE2EDuration="2.030796423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:02.027778619 +0000 UTC m=+3766.503498231" watchObservedRunningTime="2026-10-30 13:35:02.030796423 +0000 UTC m=+3766.506516033"
Oct 30 13:35:02.988710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:02.988664    4993 generic.go:356] "Generic (PLEG): container finished" podID="e3089258-552e-40b5-82a0-d4b7a96e5b58" containerID="648a7e47261ab2d24b213aa443dd4024a71c36333e9aab694518b40e5e94da5c" exitCode=0
Oct 30 13:35:02.989199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:02.988819    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72" event={"ID":"e3089258-552e-40b5-82a0-d4b7a96e5b58","Type":"ContainerDied","Data":"648a7e47261ab2d24b213aa443dd4024a71c36333e9aab694518b40e5e94da5c"}
Oct 30 13:35:04.066088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.066023    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"]
Oct 30 13:35:04.077547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.077486    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f84fc84-s5t9z"]
Oct 30 13:35:04.343884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.343843    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:04.395858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.395816    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-openshift-ca\") pod \"e3089258-552e-40b5-82a0-d4b7a96e5b58\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") "
Oct 30 13:35:04.396617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.396592    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3089258-552e-40b5-82a0-d4b7a96e5b58-kube-api-access-j8h6f\" (UniqueName: \"kubernetes.io/projected/e3089258-552e-40b5-82a0-d4b7a96e5b58-kube-api-access-j8h6f\") pod \"e3089258-552e-40b5-82a0-d4b7a96e5b58\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") "
Oct 30 13:35:04.396820 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.396804    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-xks-ca\") pod \"e3089258-552e-40b5-82a0-d4b7a96e5b58\" (UID: \"e3089258-552e-40b5-82a0-d4b7a96e5b58\") "
Oct 30 13:35:04.397898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.396480    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-openshift-ca" pod "e3089258-552e-40b5-82a0-d4b7a96e5b58" (UID: "e3089258-552e-40b5-82a0-d4b7a96e5b58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:04.400984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.400958    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3089258-552e-40b5-82a0-d4b7a96e5b58-kube-api-access-j8h6f" pod "e3089258-552e-40b5-82a0-d4b7a96e5b58" (UID: "e3089258-552e-40b5-82a0-d4b7a96e5b58"). InnerVolumeSpecName "kube-api-access-j8h6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:04.403138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.403055    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-xks-ca" pod "e3089258-552e-40b5-82a0-d4b7a96e5b58" (UID: "e3089258-552e-40b5-82a0-d4b7a96e5b58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:04.559022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.525113    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:04.559022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.525165    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8h6f\" (UniqueName: \"kubernetes.io/projected/e3089258-552e-40b5-82a0-d4b7a96e5b58-kube-api-access-j8h6f\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:04.559022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:04.525184    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3089258-552e-40b5-82a0-d4b7a96e5b58-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:05.012025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:05.011997    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"
Oct 30 13:35:05.012330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:05.012018    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72" event={"ID":"e3089258-552e-40b5-82a0-d4b7a96e5b58","Type":"ContainerDied","Data":"32c3bc3fa3264b7e7a9e64813c315a205797c3915e2026a53cd6b31738591bb4"}
Oct 30 13:35:05.012330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:05.012071    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c3bc3fa3264b7e7a9e64813c315a205797c3915e2026a53cd6b31738591bb4"
Oct 30 13:35:05.694313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:05.694268    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986a714d-458c-47dc-b878-79c74ffa9979" path="/var/lib/kubelet/pods/986a714d-458c-47dc-b878-79c74ffa9979/volumes"
Oct 30 13:35:07.181631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.181594    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"]
Oct 30 13:35:07.182321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.182301    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e3089258-552e-40b5-82a0-d4b7a96e5b58" containerName="revoke-keys"
Oct 30 13:35:07.182321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.182321    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3089258-552e-40b5-82a0-d4b7a96e5b58" containerName="revoke-keys"
Oct 30 13:35:07.182462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.182450    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3089258-552e-40b5-82a0-d4b7a96e5b58" containerName="revoke-keys"
Oct 30 13:35:07.185340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.185318    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.226615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.226570    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:35:07.229611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.227091    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:35:07.229611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.227500    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:35:07.230718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.230692    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:35:07.233967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.233926    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"]
Oct 30 13:35:07.300192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.300139    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.300476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.300410    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.300555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.300495    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhpv\" (UniqueName: \"kubernetes.io/projected/b7302120-3f5a-4fdf-ae46-685d1394dd41-kube-api-access-sbhpv\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.419693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.419630    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.420070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.420045    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhpv\" (UniqueName: \"kubernetes.io/projected/b7302120-3f5a-4fdf-ae46-685d1394dd41-kube-api-access-sbhpv\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.420401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.420383    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.421827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.421538    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.424767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.424724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.445703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.445635    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhpv\" (UniqueName: \"kubernetes.io/projected/b7302120-3f5a-4fdf-ae46-685d1394dd41-kube-api-access-sbhpv\") pod \"maas-api-revoke-sub-fd907d5e-t88sg\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.503762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.503718    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:07.829996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:07.829968    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"]
Oct 30 13:35:07.831759 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:35:07.831722    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7302120_3f5a_4fdf_ae46_685d1394dd41.slice/crio-8eb47670d92264f6b5a9abd18f0be4f0afa89218842b8fd65888137aa8e09f5e WatchSource:0}: Error finding container 8eb47670d92264f6b5a9abd18f0be4f0afa89218842b8fd65888137aa8e09f5e: Status 404 returned error can't find the container with id 8eb47670d92264f6b5a9abd18f0be4f0afa89218842b8fd65888137aa8e09f5e
Oct 30 13:35:08.043594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:08.043552    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg" event={"ID":"b7302120-3f5a-4fdf-ae46-685d1394dd41","Type":"ContainerStarted","Data":"8eb47670d92264f6b5a9abd18f0be4f0afa89218842b8fd65888137aa8e09f5e"}
Oct 30 13:35:08.574435 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:08.574359    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:35:09.166739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:09.166640    4993 generic.go:356] "Generic (PLEG): container finished" podID="b7302120-3f5a-4fdf-ae46-685d1394dd41" containerID="a5ed7097fffa465a3d54f2b0077aaaabf788633435c30256d4102b907d0ef20b" exitCode=0
Oct 30 13:35:09.167232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:09.167190    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg" event={"ID":"b7302120-3f5a-4fdf-ae46-685d1394dd41","Type":"ContainerDied","Data":"a5ed7097fffa465a3d54f2b0077aaaabf788633435c30256d4102b907d0ef20b"}
Oct 30 13:35:10.180604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.180510    4993 generic.go:356] "Generic (PLEG): container finished" podID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerID="255cf86de66be1b33cc2ba702c6e5b108f9c02f9be9c4db62b4e7573b760d94f" exitCode=0
Oct 30 13:35:10.180604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.180578    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" event={"ID":"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9","Type":"ContainerDied","Data":"255cf86de66be1b33cc2ba702c6e5b108f9c02f9be9c4db62b4e7573b760d94f"}
Oct 30 13:35:10.181622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.181368    4993 scope.go:122] "RemoveContainer" containerID="255cf86de66be1b33cc2ba702c6e5b108f9c02f9be9c4db62b4e7573b760d94f"
Oct 30 13:35:10.183617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.183572    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:10.183782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.183676    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:10.183782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.183718    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:10.458621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.458537    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:10.592250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.591670    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7302120-3f5a-4fdf-ae46-685d1394dd41-kube-api-access-sbhpv\" (UniqueName: \"kubernetes.io/projected/b7302120-3f5a-4fdf-ae46-685d1394dd41-kube-api-access-sbhpv\") pod \"b7302120-3f5a-4fdf-ae46-685d1394dd41\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") "
Oct 30 13:35:10.592250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.591944    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-openshift-ca\") pod \"b7302120-3f5a-4fdf-ae46-685d1394dd41\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") "
Oct 30 13:35:10.592250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.591998    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-xks-ca\") pod \"b7302120-3f5a-4fdf-ae46-685d1394dd41\" (UID: \"b7302120-3f5a-4fdf-ae46-685d1394dd41\") "
Oct 30 13:35:10.593692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.593659    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-openshift-ca" pod "b7302120-3f5a-4fdf-ae46-685d1394dd41" (UID: "b7302120-3f5a-4fdf-ae46-685d1394dd41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:10.603754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.603710    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-xks-ca" pod "b7302120-3f5a-4fdf-ae46-685d1394dd41" (UID: "b7302120-3f5a-4fdf-ae46-685d1394dd41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:10.606470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.606430    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7302120-3f5a-4fdf-ae46-685d1394dd41-kube-api-access-sbhpv" pod "b7302120-3f5a-4fdf-ae46-685d1394dd41" (UID: "b7302120-3f5a-4fdf-ae46-685d1394dd41"). InnerVolumeSpecName "kube-api-access-sbhpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:10.694116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.694070    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:10.694116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.694118    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7302120-3f5a-4fdf-ae46-685d1394dd41-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:10.694365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:10.694133    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhpv\" (UniqueName: \"kubernetes.io/projected/b7302120-3f5a-4fdf-ae46-685d1394dd41-kube-api-access-sbhpv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:11.196043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.196003    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" event={"ID":"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9","Type":"ContainerStarted","Data":"bd39b584f5621c755913a71aa7aa14d14ddeb82cf201406323babb186d373af6"}
Oct 30 13:35:11.198001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.197978    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"
Oct 30 13:35:11.198160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.198060    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg" event={"ID":"b7302120-3f5a-4fdf-ae46-685d1394dd41","Type":"ContainerDied","Data":"8eb47670d92264f6b5a9abd18f0be4f0afa89218842b8fd65888137aa8e09f5e"}
Oct 30 13:35:11.198160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.198122    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb47670d92264f6b5a9abd18f0be4f0afa89218842b8fd65888137aa8e09f5e"
Oct 30 13:35:11.250615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.250582    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:35:11.251292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.251261    4993 patch_prober.go:29] interesting pod/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 13:35:11.251439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.251321    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 13:35:11.430606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.430562    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"]
Oct 30 13:35:11.431192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.431175    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b7302120-3f5a-4fdf-ae46-685d1394dd41" containerName="revoke-keys"
Oct 30 13:35:11.431255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.431192    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7302120-3f5a-4fdf-ae46-685d1394dd41" containerName="revoke-keys"
Oct 30 13:35:11.431341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.431331    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7302120-3f5a-4fdf-ae46-685d1394dd41" containerName="revoke-keys"
Oct 30 13:35:11.434027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.434008    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.452127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.452051    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:35:11.452127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.452074    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:35:11.464244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.461509    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:35:11.476233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.472733    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:35:11.489232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.487441    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"]
Oct 30 13:35:11.612616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.612567    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkr6\" (UniqueName: \"kubernetes.io/projected/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-kube-api-access-wgkr6\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.612820 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.612775    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.612820 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.612815    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.714826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.714720    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.714826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.714781    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.715259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.715198    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkr6\" (UniqueName: \"kubernetes.io/projected/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-kube-api-access-wgkr6\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.715529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.715509    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.717488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.717459    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.738982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.738939    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkr6\" (UniqueName: \"kubernetes.io/projected/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-kube-api-access-wgkr6\") pod \"maas-api-revoke-sub-ddfbe4e1-8mr84\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:11.747815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:11.747777    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:12.088360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:12.088326    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"]
Oct 30 13:35:12.089584 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:35:12.089552    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c06df3_073d_49ab_bd94_f4c9e41e6f56.slice/crio-2dff2cee44d566ad35d93369208da77e4b60f98965d1ff199a495fe6807439bf WatchSource:0}: Error finding container 2dff2cee44d566ad35d93369208da77e4b60f98965d1ff199a495fe6807439bf: Status 404 returned error can't find the container with id 2dff2cee44d566ad35d93369208da77e4b60f98965d1ff199a495fe6807439bf
Oct 30 13:35:12.213100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:12.213059    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84" event={"ID":"f9c06df3-073d-49ab-bd94-f4c9e41e6f56","Type":"ContainerStarted","Data":"2dff2cee44d566ad35d93369208da77e4b60f98965d1ff199a495fe6807439bf"}
Oct 30 13:35:12.251669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:12.251628    4993 patch_prober.go:29] interesting pod/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 13:35:12.251869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:12.251693    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 13:35:13.223245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:13.223177    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84" event={"ID":"f9c06df3-073d-49ab-bd94-f4c9e41e6f56","Type":"ContainerStarted","Data":"f29dab413edcd80b978f262323de7528bd87b0e7b7f04baddff52fe73ab64396"}
Oct 30 13:35:13.251249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:13.251166    4993 patch_prober.go:29] interesting pod/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 13:35:13.251249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:13.251239    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 13:35:13.285946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:13.285865    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84" podStartSLOduration=2.2858402780000002 podStartE2EDuration="2.285840278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:13.267737761 +0000 UTC m=+3777.743457370" watchObservedRunningTime="2026-10-30 13:35:13.285840278 +0000 UTC m=+3777.761559889"
Oct 30 13:35:14.115633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:14.115596    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:35:14.232670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:14.232625    4993 generic.go:356] "Generic (PLEG): container finished" podID="f9c06df3-073d-49ab-bd94-f4c9e41e6f56" containerID="f29dab413edcd80b978f262323de7528bd87b0e7b7f04baddff52fe73ab64396" exitCode=0
Oct 30 13:35:14.233102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:14.232714    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84" event={"ID":"f9c06df3-073d-49ab-bd94-f4c9e41e6f56","Type":"ContainerDied","Data":"f29dab413edcd80b978f262323de7528bd87b0e7b7f04baddff52fe73ab64396"}
Oct 30 13:35:14.251380 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:14.251339    4993 patch_prober.go:29] interesting pod/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 13:35:14.251603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:14.251411    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 13:35:15.250993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.250952    4993 patch_prober.go:29] interesting pod/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 13:35:15.251441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.251049    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 13:35:15.492578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.492491    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:15.500241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.500179    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-xks-ca\") pod \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") "
Oct 30 13:35:15.500457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.500268    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-kube-api-access-wgkr6\" (UniqueName: \"kubernetes.io/projected/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-kube-api-access-wgkr6\") pod \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") "
Oct 30 13:35:15.500457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.500330    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-openshift-ca\") pod \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\" (UID: \"f9c06df3-073d-49ab-bd94-f4c9e41e6f56\") "
Oct 30 13:35:15.501251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.501150    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-openshift-ca" pod "f9c06df3-073d-49ab-bd94-f4c9e41e6f56" (UID: "f9c06df3-073d-49ab-bd94-f4c9e41e6f56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:15.503029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.502995    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-xks-ca" pod "f9c06df3-073d-49ab-bd94-f4c9e41e6f56" (UID: "f9c06df3-073d-49ab-bd94-f4c9e41e6f56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:15.503352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.503326    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-kube-api-access-wgkr6" pod "f9c06df3-073d-49ab-bd94-f4c9e41e6f56" (UID: "f9c06df3-073d-49ab-bd94-f4c9e41e6f56"). InnerVolumeSpecName "kube-api-access-wgkr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:15.602216 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.602151    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.602216 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.602191    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgkr6\" (UniqueName: \"kubernetes.io/projected/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-kube-api-access-wgkr6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.602481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:15.602202    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c06df3-073d-49ab-bd94-f4c9e41e6f56-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:16.272925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:16.272872    4993 patch_prober.go:29] interesting pod/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 13:35:16.273626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:16.273564    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 13:35:16.294881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:16.294837    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"
Oct 30 13:35:16.294881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:16.294859    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84" event={"ID":"f9c06df3-073d-49ab-bd94-f4c9e41e6f56","Type":"ContainerDied","Data":"2dff2cee44d566ad35d93369208da77e4b60f98965d1ff199a495fe6807439bf"}
Oct 30 13:35:16.295086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:16.294949    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dff2cee44d566ad35d93369208da77e4b60f98965d1ff199a495fe6807439bf"
Oct 30 13:35:17.256652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:17.256614    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:35:17.258009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:17.257982    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:35:20.903320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:20.903285    4993 scope.go:122] "RemoveContainer" containerID="52ff4f7d402bcb131e16c86beeb2650335b087ece38230bbbc73600b8d3f092b"
Oct 30 13:35:20.914762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:20.914735    4993 scope.go:122] "RemoveContainer" containerID="d1d33a722992b151a103dd17b03013243d668b0c4333e3b49fcd264c8fb1a8e2"
Oct 30 13:35:20.925324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:20.925294    4993 scope.go:122] "RemoveContainer" containerID="5846f3cc599959620a701bc26d69eed5e06fdbc396cfe1d0f8f92d1fa288a5e4"
Oct 30 13:35:20.942992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:20.942951    4993 scope.go:122] "RemoveContainer" containerID="5e30eaae52ab867cd8fb535b40b42778e0f29e03fafd5c3487818425bfa97f6a"
Oct 30 13:35:20.957123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:20.957096    4993 scope.go:122] "RemoveContainer" containerID="73b454ff51323a2df1e7c0bb68dec2894ca7f8655282f475073bfc99d2d000ec"
Oct 30 13:35:20.967269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:20.967241    4993 scope.go:122] "RemoveContainer" containerID="c9ec658fa0e90f5956d794b3d53d4a657393c50c00f0e8335f435589495c1a54"
Oct 30 13:35:20.977179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:20.977151    4993 scope.go:122] "RemoveContainer" containerID="8a36f6b0b31905748cfd51b70b45441b59e0ac64b39cdc249590a87765e5e669"
Oct 30 13:35:20.987235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:20.987198    4993 scope.go:122] "RemoveContainer" containerID="7663b4e0180ae9fc8ff9e9eeed3f0a0aa394b9bf3bd7506064441b757ed69c2f"
Oct 30 13:35:23.857305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:23.857263    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"]
Oct 30 13:35:23.858120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:23.858066    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p" podUID="4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6" containerName="sim" containerID="cri-o://1115374a94adf88c739df1f702769afd6ff9aca10752622df1c07fac4d875c0a" gracePeriod=30
Oct 30 13:35:24.449245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.449170    4993 generic.go:356] "Generic (PLEG): container finished" podID="4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6" containerID="1115374a94adf88c739df1f702769afd6ff9aca10752622df1c07fac4d875c0a" exitCode=0
Oct 30 13:35:24.449460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.449248    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p" event={"ID":"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6","Type":"ContainerDied","Data":"1115374a94adf88c739df1f702769afd6ff9aca10752622df1c07fac4d875c0a"}
Oct 30 13:35:24.521096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.521062    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"]
Oct 30 13:35:24.522193 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.522167    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f9c06df3-073d-49ab-bd94-f4c9e41e6f56" containerName="revoke-keys"
Oct 30 13:35:24.522301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.522194    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c06df3-073d-49ab-bd94-f4c9e41e6f56" containerName="revoke-keys"
Oct 30 13:35:24.522401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.522386    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c06df3-073d-49ab-bd94-f4c9e41e6f56" containerName="revoke-keys"
Oct 30 13:35:24.525773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.525750    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.551932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.551902    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"]
Oct 30 13:35:24.559777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.559747    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"agc-cred-adafb933cb9f\""
Oct 30 13:35:24.560077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.560052    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-e2e--8217e17a-dockercfg-v72th\""
Oct 30 13:35:24.563458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.563435    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"routing-overlay\""
Oct 30 13:35:24.563591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.563468    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1957dc\""
Oct 30 13:35:24.563591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.563508    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"openshift-service-ca.crt\""
Oct 30 13:35:24.569540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.569520    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"kube-root-ca.crt\""
Oct 30 13:35:24.619118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.618754    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9rb\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-kube-api-access-zv9rb\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.619118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.618863    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.619118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.618915    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.619118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.619048    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.721366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.721263    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.721546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.721482    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9rb\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-kube-api-access-zv9rb\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.721620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.721568    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.721683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.721622    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.722164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.722135    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.722347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.722324    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.724221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.724191    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.746202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.746174    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9rb\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-kube-api-access-zv9rb\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.780776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.780738    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:35:24.841553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.841511    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:24.927196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.926026    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-kube-api-access-c6c6l\" (UniqueName: \"kubernetes.io/projected/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-kube-api-access-c6c6l\") pod \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\" (UID: \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\") "
Oct 30 13:35:24.927196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.926120    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-tls\" (UniqueName: \"kubernetes.io/secret/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-tls\") pod \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\" (UID: \"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6\") "
Oct 30 13:35:24.931440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.931391    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-kube-api-access-c6c6l" pod "4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6" (UID: "4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6"). InnerVolumeSpecName "kube-api-access-c6c6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:24.931710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:24.931683    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-tls" pod "4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6" (UID: "4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:25.028680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.028650    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6c6l\" (UniqueName: \"kubernetes.io/projected/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-kube-api-access-c6c6l\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:25.028680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.028680    4993 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6-tls\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:25.158758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.158730    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"]
Oct 30 13:35:25.161614 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:35:25.161579    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afc454c_b0c7_493e_81b6_b53b784f5778.slice/crio-71f78f4c2b0bc1b54a6612c55161caab4a3b9ee142e97cfcf226e61aca3e55fa WatchSource:0}: Error finding container 71f78f4c2b0bc1b54a6612c55161caab4a3b9ee142e97cfcf226e61aca3e55fa: Status 404 returned error can't find the container with id 71f78f4c2b0bc1b54a6612c55161caab4a3b9ee142e97cfcf226e61aca3e55fa
Oct 30 13:35:25.479660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.479611    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" event={"ID":"1afc454c-b0c7-493e-81b6-b53b784f5778","Type":"ContainerStarted","Data":"e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a"}
Oct 30 13:35:25.479889 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.479722    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:25.479889 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.479747    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" event={"ID":"1afc454c-b0c7-493e-81b6-b53b784f5778","Type":"ContainerStarted","Data":"71f78f4c2b0bc1b54a6612c55161caab4a3b9ee142e97cfcf226e61aca3e55fa"}
Oct 30 13:35:25.481529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.481498    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"
Oct 30 13:35:25.481529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.481513    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p" event={"ID":"4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6","Type":"ContainerDied","Data":"12abd604760d1b43da23f0a5c52d7381b5a697ff93688fc00380210b65f78f97"}
Oct 30 13:35:25.481686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.481566    4993 scope.go:122] "RemoveContainer" containerID="1115374a94adf88c739df1f702769afd6ff9aca10752622df1c07fac4d875c0a"
Oct 30 13:35:25.528859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.528791    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" podStartSLOduration=1.528770947 podStartE2EDuration="1.528770947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:25.525009792 +0000 UTC m=+3790.000729402" watchObservedRunningTime="2026-10-30 13:35:25.528770947 +0000 UTC m=+3790.004490557"
Oct 30 13:35:25.573529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.573488    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"]
Oct 30 13:35:25.580763 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.580726    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-99f4a4/e2e-usage-sim-ee9e02-d5fd59d59-t2z4p"]
Oct 30 13:35:25.639456 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:25.639418    4993 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-adafb933cb9f: secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:25.639456 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:25.639451    4993 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh: secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:25.639731 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:25.639512    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials podName:1afc454c-b0c7-493e-81b6-b53b784f5778 nodeName:}" failed. No retries permitted until 2026-10-30 13:35:26.139492088 +0000 UTC m=+3790.615211680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778") : secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:25.725551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:25.724846    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6" path="/var/lib/kubelet/pods/4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6/volumes"
Oct 30 13:35:26.017964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:26.017920    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"]
Oct 30 13:35:26.192979 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:26.192934    4993 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-adafb933cb9f: secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:26.192979 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:26.192973    4993 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh: secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:26.193292 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:26.193043    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials podName:1afc454c-b0c7-493e-81b6-b53b784f5778 nodeName:}" failed. No retries permitted until 2026-10-30 13:35:27.193022275 +0000 UTC m=+3791.668741876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778") : secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:26.612241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:26.605268    4993 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" secret="" err="secret \"payload-processing-external-model-e2e--8217e17a-dockercfg-v72th\" not found"
Oct 30 13:35:26.639949 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:26.637182    4993 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-plugins-e2e-extmod-a-1957dc: configmap "payload-processing-external-model-plugins-e2e-extmod-a-1957dc" not found
Oct 30 13:35:26.639949 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:26.637276    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume podName:1afc454c-b0c7-493e-81b6-b53b784f5778 nodeName:}" failed. No retries permitted until 2026-10-30 13:35:27.137254887 +0000 UTC m=+3791.612974476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-1957dc" not found
Oct 30 13:35:27.144327 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:27.144291    4993 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-plugins-e2e-extmod-a-1957dc: configmap "payload-processing-external-model-plugins-e2e-extmod-a-1957dc" not found
Oct 30 13:35:27.144755 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:27.144364    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume podName:1afc454c-b0c7-493e-81b6-b53b784f5778 nodeName:}" failed. No retries permitted until 2026-10-30 13:35:28.144348271 +0000 UTC m=+3792.620067876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-1957dc" not found
Oct 30 13:35:27.245398 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:27.245358    4993 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-adafb933cb9f: secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:27.245398 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:27.245392    4993 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh: secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:27.245632 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:27.245447    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials podName:1afc454c-b0c7-493e-81b6-b53b784f5778 nodeName:}" failed. No retries permitted until 2026-10-30 13:35:29.245431128 +0000 UTC m=+3793.721150736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778") : secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:28.096255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.091027    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"]
Oct 30 13:35:28.096255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.092032    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6" containerName="sim"
Oct 30 13:35:28.096255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.092056    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6" containerName="sim"
Oct 30 13:35:28.096255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.092315    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e06be5f-7e1e-4060-9e9a-5d5a893ebfe6" containerName="sim"
Oct 30 13:35:28.096883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.096860    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.140340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.133640    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"]
Oct 30 13:35:28.155910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.155875    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-025c7f-openshift-default-dockercfg-ztrnc\""
Oct 30 13:35:28.233363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.233314    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-podinfo\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.233656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.233636    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-certs\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.233809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.233794    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.233979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.233962    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-data\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.234154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.234134    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-credential-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.238228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.238181    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-envoy\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.240288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.240259    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istiod-ca-cert\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.240579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.240559    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2mt\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-kube-api-access-9x2mt\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.240790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.240775    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-token\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.241094 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:28.239298    4993 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-plugins-e2e-extmod-a-1957dc: configmap "payload-processing-external-model-plugins-e2e-extmod-a-1957dc" not found
Oct 30 13:35:28.241299 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:28.241286    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume podName:1afc454c-b0c7-493e-81b6-b53b784f5778 nodeName:}" failed. No retries permitted until 2026-10-30 13:35:30.241246384 +0000 UTC m=+3794.716965976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-1957dc" not found
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.372675    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-envoy\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.372770    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istiod-ca-cert\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.372819    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2mt\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-kube-api-access-9x2mt\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.372906    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-token\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.372948    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-podinfo\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.372991    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-certs\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.373041    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.373078    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-data\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.373135    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-credential-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.373640    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-credential-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.381518    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istiod-ca-cert\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.382066    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.382322    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-certs\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.389286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.382623    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-data\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.429900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.419662    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-envoy\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.452864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.452830    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-token\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.481519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.481369    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-podinfo\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.492315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.483864    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2mt\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-kube-api-access-9x2mt\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:28.725777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:28.725680    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:29.411239 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:29.383661    4993 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-adafb933cb9f: secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:29.419535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:29.419482    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:29.419937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:29.419903    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:29.420163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:29.420143    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:29.421117 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:29.419130    4993 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh: secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:29.422591 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:29.422570    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials podName:1afc454c-b0c7-493e-81b6-b53b784f5778 nodeName:}" failed. No retries permitted until 2026-10-30 13:35:33.421684455 +0000 UTC m=+3797.897404060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778") : secret "agc-cred-adafb933cb9f" not found
Oct 30 13:35:29.451000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:29.450952    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"]
Oct 30 13:35:29.696231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:29.695600    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" event={"ID":"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30","Type":"ContainerStarted","Data":"bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873"}
Oct 30 13:35:29.696231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:29.695658    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" event={"ID":"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30","Type":"ContainerStarted","Data":"015de7fb2fae1906e8ab6ffeafdd086db9f48f268997c395d3fc8f17798fb8e6"}
Oct 30 13:35:29.728459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:29.728409    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:29.741562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:29.741500    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podStartSLOduration=1.7414842259999999 podStartE2EDuration="1.741484226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:29.740001173 +0000 UTC m=+3794.215720783" watchObservedRunningTime="2026-10-30 13:35:29.741484226 +0000 UTC m=+3794.217203835"
Oct 30 13:35:30.310892 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:30.310848    4993 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-plugins-e2e-extmod-a-1957dc: configmap "payload-processing-external-model-plugins-e2e-extmod-a-1957dc" not found
Oct 30 13:35:30.311122 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:30.310944    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume podName:1afc454c-b0c7-493e-81b6-b53b784f5778 nodeName:}" failed. No retries permitted until 2026-10-30 13:35:34.310922435 +0000 UTC m=+3798.786642026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-1957dc" not found
Oct 30 13:35:30.734828 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:30.734794    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:31.708277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:31.708240    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:31.709302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:31.709273    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:35:32.648229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:32.647791    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" podUID="1afc454c-b0c7-493e-81b6-b53b784f5778" containerName="payload-processing" containerID="cri-o://e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a" gracePeriod=25
Oct 30 13:35:32.961056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:32.961024    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:33.078319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.078279    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume\") pod \"1afc454c-b0c7-493e-81b6-b53b784f5778\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") "
Oct 30 13:35:33.078509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.078346    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials\") pod \"1afc454c-b0c7-493e-81b6-b53b784f5778\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") "
Oct 30 13:35:33.078575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.078534    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-kube-api-access-zv9rb\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-kube-api-access-zv9rb\") pod \"1afc454c-b0c7-493e-81b6-b53b784f5778\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") "
Oct 30 13:35:33.078655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.078622    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-routing-overlay-volume\") pod \"1afc454c-b0c7-493e-81b6-b53b784f5778\" (UID: \"1afc454c-b0c7-493e-81b6-b53b784f5778\") "
Oct 30 13:35:33.079738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.079707    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-routing-overlay-volume" pod "1afc454c-b0c7-493e-81b6-b53b784f5778" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:33.080097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.080077    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume" pod "1afc454c-b0c7-493e-81b6-b53b784f5778" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:33.083119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.083092    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials" pod "1afc454c-b0c7-493e-81b6-b53b784f5778" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:33.083297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.083134    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-kube-api-access-zv9rb" pod "1afc454c-b0c7-493e-81b6-b53b784f5778" (UID: "1afc454c-b0c7-493e-81b6-b53b784f5778"). InnerVolumeSpecName "kube-api-access-zv9rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:33.180764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.180655    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv9rb\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-kube-api-access-zv9rb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:33.180764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.180700    4993 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-routing-overlay-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:33.180764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.180713    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afc454c-b0c7-493e-81b6-b53b784f5778-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:33.180764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.180726    4993 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1afc454c-b0c7-493e-81b6-b53b784f5778-provider-credentials\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:35:33.252410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.252369    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"]
Oct 30 13:35:33.253628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.253591    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1afc454c-b0c7-493e-81b6-b53b784f5778" containerName="payload-processing"
Oct 30 13:35:33.253628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.253624    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1afc454c-b0c7-493e-81b6-b53b784f5778" containerName="payload-processing"
Oct 30 13:35:33.253877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.253861    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1afc454c-b0c7-493e-81b6-b53b784f5778" containerName="payload-processing"
Oct 30 13:35:33.269231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.266051    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.288634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.286970    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9a93bd-openshift-default-dockercfg-mgbm5\""
Oct 30 13:35:33.296821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.296785    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"]
Oct 30 13:35:33.385822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.385771    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.386013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.385936    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.386091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.386006    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.386150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.386097    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7sd\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-kube-api-access-tn7sd\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.386257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.386239    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.386370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.386349    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.386450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.386433    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.386579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.386563    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.386631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.386605    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.488183    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.488746    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.488828    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.488916    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.488947    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.489045    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7sd\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-kube-api-access-tn7sd\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.489094    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.489174    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.489269    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.490232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.489737    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.499768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.495094    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.500530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.500493    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.500727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.500702    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.501031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.501007    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.520332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.517504    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.523359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.522076    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.546231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.545502    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.548034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.547953    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7sd\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-kube-api-access-tn7sd\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.612234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.609883    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:33.921880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.921844    4993 generic.go:356] "Generic (PLEG): container finished" podID="1afc454c-b0c7-493e-81b6-b53b784f5778" containerID="e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a" exitCode=0
Oct 30 13:35:33.923995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.923971    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"
Oct 30 13:35:33.927477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.926963    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" event={"ID":"1afc454c-b0c7-493e-81b6-b53b784f5778","Type":"ContainerDied","Data":"e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a"}
Oct 30 13:35:33.927477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.927026    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh" event={"ID":"1afc454c-b0c7-493e-81b6-b53b784f5778","Type":"ContainerDied","Data":"71f78f4c2b0bc1b54a6612c55161caab4a3b9ee142e97cfcf226e61aca3e55fa"}
Oct 30 13:35:33.927477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.927054    4993 scope.go:122] "RemoveContainer" containerID="e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a"
Oct 30 13:35:33.984198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.983471    4993 scope.go:122] "RemoveContainer" containerID="e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a"
Oct 30 13:35:33.984725 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:33.984642    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a\": container with ID starting with e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a not found: ID does not exist" containerID="e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a"
Oct 30 13:35:33.984725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:33.984687    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a"} err="failed to get container status \"e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a\": rpc error: code = NotFound desc = could not find container \"e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a\": container with ID starting with e2eb810b0a00d70d6aafcd54f2645a1c8cb2f4a8570aa14c4b77ddfa9860bc4a not found: ID does not exist"
Oct 30 13:35:34.042353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.042265    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"]
Oct 30 13:35:34.063551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.063511    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dgjpqh"]
Oct 30 13:35:34.551181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.549308    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:34.551181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.549449    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:34.551181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.549499    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:34.705793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.705742    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"]
Oct 30 13:35:34.935054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.935005    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" event={"ID":"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3","Type":"ContainerStarted","Data":"ac9f5c3d7811277e72cd240634adc79a58747b47ff25350ee37fede2f4a5cca1"}
Oct 30 13:35:34.935520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.935097    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" event={"ID":"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3","Type":"ContainerStarted","Data":"c36896160b2468637c1c4adbb76cccec870c133b3fbf60188e843332ec12e34b"}
Oct 30 13:35:34.992917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:34.992106    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podStartSLOduration=1.99208276 podStartE2EDuration="1.99208276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:34.987909107 +0000 UTC m=+3799.463628730" watchObservedRunningTime="2026-10-30 13:35:34.99208276 +0000 UTC m=+3799.467802370"
Oct 30 13:35:35.610727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.610685    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:35.617479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.617447    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:35.666229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.665990    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"]
Oct 30 13:35:35.680299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.672332    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.697111 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.697080    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-09ad1e-openshift-default-dockercfg-h2w8d\""
Oct 30 13:35:35.699240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.699175    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1afc454c-b0c7-493e-81b6-b53b784f5778" path="/var/lib/kubelet/pods/1afc454c-b0c7-493e-81b6-b53b784f5778/volumes"
Oct 30 13:35:35.714253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.713127    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"]
Oct 30 13:35:35.806760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.806715    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-envoy\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.806950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.806778    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istiod-ca-cert\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.806950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.806900    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-token\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.807051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.807025    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7v4\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-kube-api-access-br7v4\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.807120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.807105    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-credential-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.807219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.807185    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.807270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.807253    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-certs\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.807382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.807368    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-data\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.807523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.807504    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-podinfo\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912335    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-envoy\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912426    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istiod-ca-cert\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912466    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-token\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912532    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7v4\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-kube-api-access-br7v4\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912591    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-credential-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912633    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912672    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-certs\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912752    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-data\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.913539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.912849    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-podinfo\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.914732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.914691    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.914910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.914792    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istiod-ca-cert\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.915059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.915027    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-certs\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.915467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.915434    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-credential-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.915909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.915876    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-data\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.966249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.966191    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-envoy\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:35.981937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:35.974433    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-podinfo\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:36.011751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.011718    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:36.058230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.047856    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-token\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:36.058230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.048555    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7v4\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-kube-api-access-br7v4\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:36.070237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.063371    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:35:36.298239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.295883    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:36.749523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.749481    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"]
Oct 30 13:35:36.766741 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:35:36.766702    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650859ec_868e_46f6_8a0a_1ab7c5ab14cc.slice/crio-6bdaa0427180c7f8bf0f54509217a8f0e11b13eac937e525fe3125d750605185 WatchSource:0}: Error finding container 6bdaa0427180c7f8bf0f54509217a8f0e11b13eac937e525fe3125d750605185: Status 404 returned error can't find the container with id 6bdaa0427180c7f8bf0f54509217a8f0e11b13eac937e525fe3125d750605185
Oct 30 13:35:36.770926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.770887    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:36.771061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.770981    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:36.771061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:36.771034    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:37.025191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:37.025076    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" event={"ID":"650859ec-868e-46f6-8a0a-1ab7c5ab14cc","Type":"ContainerStarted","Data":"27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88"}
Oct 30 13:35:37.025191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:37.025140    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" event={"ID":"650859ec-868e-46f6-8a0a-1ab7c5ab14cc","Type":"ContainerStarted","Data":"6bdaa0427180c7f8bf0f54509217a8f0e11b13eac937e525fe3125d750605185"}
Oct 30 13:35:37.076229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:37.073873    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podStartSLOduration=2.073853382 podStartE2EDuration="2.073853382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:37.070287793 +0000 UTC m=+3801.546007413" watchObservedRunningTime="2026-10-30 13:35:37.073853382 +0000 UTC m=+3801.549572994"
Oct 30 13:35:37.304160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:37.304068    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:37.304543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:37.304517    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:35:37.304729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:37.304705    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 13:35:38.308039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:38.308004    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:38.570289 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:35:38.570185    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:35:39.112716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:39.112681    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:39.113650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:39.113626    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:35:44.078232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:44.070509    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 13:35:44.097232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:44.091993    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:44.126708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:44.126656    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:44.132610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:44.132559    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:44.218417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:44.218357    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:44.335306 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:44.331515    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:44.503233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:44.502759    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:45.044036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:45.043987    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"]
Oct 30 13:35:45.049758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:45.049701    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994c677-mbccl"]
Oct 30 13:35:45.782817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:45.782775    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7367a9b-35e7-4115-93d8-de704cc5a5c6" path="/var/lib/kubelet/pods/d7367a9b-35e7-4115-93d8-de704cc5a5c6/volumes"
Oct 30 13:35:57.533991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:57.533563    4993 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": read tcp 10.132.0.2:32818->10.132.1.36:15021: read: connection reset by peer" start-of-body=
Oct 30 13:35:57.533991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:57.533713    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": read tcp 10.132.0.2:32818->10.132.1.36:15021: read: connection reset by peer"
Oct 30 13:35:57.882940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:57.882841    4993 generic.go:356] "Generic (PLEG): container finished" podID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerID="a8ceb1b41a76b585a16af3c45d43fc90dda8f5fdfbcad8287f25f21021694c0c" exitCode=0
Oct 30 13:35:57.883126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:57.882931    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" event={"ID":"4a44edeb-97be-4ef5-878c-7de3abd1bedd","Type":"ContainerDied","Data":"a8ceb1b41a76b585a16af3c45d43fc90dda8f5fdfbcad8287f25f21021694c0c"}
Oct 30 13:35:57.883467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:57.883448    4993 scope.go:122] "RemoveContainer" containerID="a8ceb1b41a76b585a16af3c45d43fc90dda8f5fdfbcad8287f25f21021694c0c"
Oct 30 13:35:57.885566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:57.885536    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:57.885677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:57.885610    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:57.885677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:57.885643    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:58.894446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:58.894407    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" event={"ID":"4a44edeb-97be-4ef5-878c-7de3abd1bedd","Type":"ContainerStarted","Data":"7017126add71196eef540c2924b20960e5247d57e1ba6cfc6a336cb438ef19b8"}
Oct 30 13:35:59.728299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:59.728263    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:35:59.728489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:59.728432    4993 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:35:59.728489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:35:59.728480    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 30 13:36:00.728465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:00.728425    4993 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:36:00.728905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:00.728497    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 30 13:36:01.729108 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:01.729071    4993 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:36:01.729794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:01.729128    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 30 13:36:02.733235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:02.730334    4993 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:36:02.733235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:02.730413    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 30 13:36:03.731999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:03.731963    4993 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:36:03.732259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:03.732017    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused"
Oct 30 13:36:04.983829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:04.983796    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:36:04.985551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:04.985527    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:36:04.988942 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:04.988736    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:36:07.037096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.037041    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"]
Oct 30 13:36:07.042738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.042713    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.081736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.074228    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b7407-openshift-default-dockercfg-99s4b\""
Oct 30 13:36:07.090769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.090692    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"]
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121225    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-data\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121319    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-podinfo\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121363    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-envoy\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121426    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-token\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121523    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-certs\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121557    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-credential-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121707    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121732    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istiod-ca-cert\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.125554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.121811    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnb5n\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-kube-api-access-dnb5n\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.176286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.140327    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9503e118-djjq6"]
Oct 30 13:36:07.176286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.156104    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"]
Oct 30 13:36:07.287288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.239343    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.287717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.287684    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istiod-ca-cert\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.313233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.287910    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnb5n\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-kube-api-access-dnb5n\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.313559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.313526    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-data\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.313782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.313761    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-podinfo\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.313935 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.313919    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-envoy\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.323233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.314091    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-token\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.332234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.323563    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-certs\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.338244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.332480    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-credential-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.353230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.323811    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-data\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.365228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.301998    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istiod-ca-cert\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.382316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.332522    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-certs\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.404321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.248802    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.404565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.332756    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-credential-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.566516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.565166    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnb5n\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-kube-api-access-dnb5n\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.566516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.566307    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-podinfo\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.566516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.566422    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-token\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.569157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.569132    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-envoy\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.696572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.696541    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:07.797507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:07.797476    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a62756c-45ea-4b59-9c74-5781f0ff4afc" path="/var/lib/kubelet/pods/8a62756c-45ea-4b59-9c74-5781f0ff4afc/volumes"
Oct 30 13:36:08.204705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:08.204612    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"]
Oct 30 13:36:08.215074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:08.215030    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:08.215225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:08.215118    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:08.215225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:08.215162    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:08.608872 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:36:08.608774    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:36:09.141488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:09.141447    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" event={"ID":"ed8cd26d-ec22-4f27-9565-26f6e066c1f8","Type":"ContainerStarted","Data":"d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7"}
Oct 30 13:36:09.141766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:09.141745    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" event={"ID":"ed8cd26d-ec22-4f27-9565-26f6e066c1f8","Type":"ContainerStarted","Data":"61a07264ea9a27083373793488ec7e4fa26ab95c435bb5e90c187da11d138da4"}
Oct 30 13:36:09.208146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:09.208079    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podStartSLOduration=3.208058026 podStartE2EDuration="3.208058026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:09.203307329 +0000 UTC m=+3833.679026942" watchObservedRunningTime="2026-10-30 13:36:09.208058026 +0000 UTC m=+3833.683777637"
Oct 30 13:36:09.698728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:09.698692    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:09.706762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:09.706734    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:10.200241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:10.197690    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:10.230230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:10.225880    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:36:12.795415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:12.795361    4993 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:36:12.795962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:12.795452    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:15.256550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:15.256496    4993 generic.go:356] "Generic (PLEG): container finished" podID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerID="445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c" exitCode=0
Oct 30 13:36:15.257012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:15.256860    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" event={"ID":"d00cfe34-6ca8-4a28-a921-968f1368fb10","Type":"ContainerDied","Data":"445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c"}
Oct 30 13:36:15.257516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:15.257488    4993 scope.go:122] "RemoveContainer" containerID="445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c"
Oct 30 13:36:15.260371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:15.260334    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:15.260495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:15.260421    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:15.260495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:15.260457    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:16.314807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:16.314762    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" event={"ID":"d00cfe34-6ca8-4a28-a921-968f1368fb10","Type":"ContainerStarted","Data":"3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94"}
Oct 30 13:36:16.470865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:16.470829    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:36:16.471364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:16.471328    4993 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 13:36:16.471589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:16.471566    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 13:36:17.461727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:17.461681    4993 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 13:36:17.462404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:17.462372    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 13:36:18.470232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:18.469666    4993 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 13:36:18.470232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:18.469737    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 13:36:19.461232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:19.460702    4993 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 13:36:19.461232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:19.460772    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 13:36:20.460681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:20.460639    4993 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 13:36:20.461350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:20.461316    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 13:36:21.080231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:21.071966    4993 scope.go:122] "RemoveContainer" containerID="a4f5959007f0a1ff0b246c1a857fc195ddd5aa0b935a6a155980eb736170b7f5"
Oct 30 13:36:21.101738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:21.101714    4993 scope.go:122] "RemoveContainer" containerID="a59478b7e2aab9213b573c04a6bb7a4f90826cf2223f5b46eb04ddfe37918aff"
Oct 30 13:36:21.115000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:21.114969    4993 scope.go:122] "RemoveContainer" containerID="948138374f3b6253e08a5f3af143209a5c1fd76fba4721260d494417422d197c"
Oct 30 13:36:21.125944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:21.125907    4993 scope.go:122] "RemoveContainer" containerID="c7670b7069e86c280c548c2959c31ace4b1944b419d99cc29334461071a17348"
Oct 30 13:36:21.473290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:21.471858    4993 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 13:36:21.473290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:21.471932    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 13:36:22.464899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:22.464860    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:36:22.465826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:22.465805    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:36:22.466572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:22.466543    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:36:32.343698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.343652    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"]
Oct 30 13:36:32.350820 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.350770    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.373282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.372960    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ba646b\"/\"default-dockercfg-67cbt\""
Oct 30 13:36:32.373282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.373126    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ba646b\"/\"test-model-ba646b-kserve-self-signed-certs\""
Oct 30 13:36:32.380727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.380692    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ba646b\"/\"kube-root-ca.crt\""
Oct 30 13:36:32.383157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.383096    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"]
Oct 30 13:36:32.462646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.462594    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90039dac-eba4-4da5-854e-ed5e146756ef-tls-certs\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.462904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.462741    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-home\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.462904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.462824    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-tmp-dir\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.462904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.462851    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-model-cache\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.462904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.462894    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq8xm\" (UniqueName: \"kubernetes.io/projected/90039dac-eba4-4da5-854e-ed5e146756ef-kube-api-access-wq8xm\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.463272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.463160    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-dshm\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.467811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.467778    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ba646b\"/\"openshift-service-ca.crt\""
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.583003    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-home\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.583095    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-tmp-dir\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.583133    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-model-cache\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.583188    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq8xm\" (UniqueName: \"kubernetes.io/projected/90039dac-eba4-4da5-854e-ed5e146756ef-kube-api-access-wq8xm\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.583340    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-dshm\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.583510    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90039dac-eba4-4da5-854e-ed5e146756ef-tls-certs\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.584426    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-model-cache\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.584690    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-home\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.586237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.584966    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-tmp-dir\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.605712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.605580    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-dshm\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.621232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.616907    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90039dac-eba4-4da5-854e-ed5e146756ef-tls-certs\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.641240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.638364    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq8xm\" (UniqueName: \"kubernetes.io/projected/90039dac-eba4-4da5-854e-ed5e146756ef-kube-api-access-wq8xm\") pod \"test-model-ba646b-kserve-56d86c4fd5-d7zsk\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") " pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:32.684235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:32.682900    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:33.539333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:33.539297    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"]
Oct 30 13:36:33.542020 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:36:33.541992    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90039dac_eba4_4da5_854e_ed5e146756ef.slice/crio-9619c7bccaaa2e39c25812c9fccfb53df5dce9834bcc74aaae66586e0d1b6a64 WatchSource:0}: Error finding container 9619c7bccaaa2e39c25812c9fccfb53df5dce9834bcc74aaae66586e0d1b6a64: Status 404 returned error can't find the container with id 9619c7bccaaa2e39c25812c9fccfb53df5dce9834bcc74aaae66586e0d1b6a64
Oct 30 13:36:34.598231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:34.597260    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk" event={"ID":"90039dac-eba4-4da5-854e-ed5e146756ef","Type":"ContainerStarted","Data":"2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b"}
Oct 30 13:36:34.598231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:34.597327    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk" event={"ID":"90039dac-eba4-4da5-854e-ed5e146756ef","Type":"ContainerStarted","Data":"9619c7bccaaa2e39c25812c9fccfb53df5dce9834bcc74aaae66586e0d1b6a64"}
Oct 30 13:36:34.642194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:34.642124    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk" podStartSLOduration=2.642110558 podStartE2EDuration="2.642110558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:34.63963993 +0000 UTC m=+3859.115359540" watchObservedRunningTime="2026-10-30 13:36:34.642110558 +0000 UTC m=+3859.117830168"
Oct 30 13:36:34.683074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:34.683023    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:38.608159 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:36:38.608109    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:36:42.684022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:42.683976    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:42.697159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:42.697126    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:36:43.693795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:36:43.693764    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:37:08.100372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:08.100307    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:08.130383 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:08.130327    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:08.131810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:08.131773    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:08.249174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:08.249118    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:08.670028 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:37:08.669955    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:37:10.254644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:10.254593    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"]
Oct 30 13:37:10.325060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:10.325020    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75dadf4-crszm"]
Oct 30 13:37:10.858232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:10.826941    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:11.731273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:11.729921    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f749cc-c785-4f0b-9a5d-2e09a27d2b4b" path="/var/lib/kubelet/pods/27f749cc-c785-4f0b-9a5d-2e09a27d2b4b/volumes"
Oct 30 13:37:11.731273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:11.730492    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:16.414841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:16.414804    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:37:16.450356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:16.450317    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:37:17.751376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:17.751329    4993 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:37:17.751814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:17.751431    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:21.231126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:21.231083    4993 scope.go:122] "RemoveContainer" containerID="2d54816471e133b138e6fd299783fa1497a946f83d167f496341a4159ee53617"
Oct 30 13:37:21.336034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:21.336007    4993 scope.go:122] "RemoveContainer" containerID="7ed555867e236db6303c37400c97a3a4d77a5e784d7405773c82483dda71efe7"
Oct 30 13:37:25.163497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:25.163440    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:37:25.164159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:25.163550    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:26.742428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.742397    4993 generic.go:356] "Generic (PLEG): container finished" podID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerID="bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873" exitCode=0
Oct 30 13:37:26.742824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.742467    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" event={"ID":"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30","Type":"ContainerDied","Data":"bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873"}
Oct 30 13:37:26.743010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.742989    4993 scope.go:122] "RemoveContainer" containerID="bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873"
Oct 30 13:37:26.744573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.744550    4993 generic.go:356] "Generic (PLEG): container finished" podID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerID="27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88" exitCode=0
Oct 30 13:37:26.744675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.744574    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" event={"ID":"650859ec-868e-46f6-8a0a-1ab7c5ab14cc","Type":"ContainerDied","Data":"27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88"}
Oct 30 13:37:26.745089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.745069    4993 scope.go:122] "RemoveContainer" containerID="27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88"
Oct 30 13:37:26.745782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.745741    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:26.745837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.745815    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:26.745876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.745846    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:26.747308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.747283    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:26.747379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.747347    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:26.747435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:26.747381    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:27.755257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:27.755190    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" event={"ID":"650859ec-868e-46f6-8a0a-1ab7c5ab14cc","Type":"ContainerStarted","Data":"70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86"}
Oct 30 13:37:27.758042 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:27.758010    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" event={"ID":"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30","Type":"ContainerStarted","Data":"45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b"}
Oct 30 13:37:28.311228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:28.303290    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:37:28.311228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:28.309420    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:37:28.311228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:28.309502    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 13:37:28.728442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:28.728405    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:37:28.729172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:28.729144    4993 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:37:28.729273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:28.729245    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 13:37:29.304694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:29.304654    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:37:29.305387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:29.305355    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 13:37:29.728726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:29.728672    4993 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:37:29.728902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:29.728777    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 13:37:30.303693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:30.303654    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:37:30.304034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:30.303985    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 13:37:30.728533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:30.728498    4993 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:37:30.729023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:30.728559    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 13:37:31.108554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.107269    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"]
Oct 30 13:37:31.158039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.157405    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488320    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488430    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xq9g\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-kube-api-access-8xq9g\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488502    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488539    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488584    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488628    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72cc892-2538-41a7-8c6a-861a7775e4d7-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488733    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488860    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.489235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.488893    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.521592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.490702    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f97234-openshift-default-dockercfg-x8qzl\""
Oct 30 13:37:31.521592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.494847    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:37:31.521592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.494895    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 13:37:31.579232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.576087    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"]
Oct 30 13:37:31.673674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.599268    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.682245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.674035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xq9g\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-kube-api-access-8xq9g\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.682640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.682602    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.683155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.683131    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.719500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.694095    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.719500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.670086    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.720161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.720127    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.743946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.743859    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72cc892-2538-41a7-8c6a-861a7775e4d7-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.744613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.744587    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.744833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.744817    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.744943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.744930    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.745579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.745554    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.745739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.733335    4993 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:37:31.745902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.745873    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 13:37:31.751947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.747067    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72cc892-2538-41a7-8c6a-861a7775e4d7-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.802327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.752138    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.802327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.752403    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.802327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.753806    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.802327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.783401    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xq9g\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-kube-api-access-8xq9g\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.802327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.788320    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:31.847229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.832949    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:37:31.954589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:31.954546    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:32.132797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:32.131749    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:32.132797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:32.132258    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 13:37:32.147682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:32.147608    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:32.320291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:32.319353    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:37:32.320291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:32.319434    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 13:37:32.808238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:32.776912    4993 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:37:32.918283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:32.916491    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 13:37:33.218230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.198705    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:33.218230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.198808    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:33.218230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.198854    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:33.248526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.248039    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"]
Oct 30 13:37:33.249436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.249289    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" event={"ID":"a72cc892-2538-41a7-8c6a-861a7775e4d7","Type":"ContainerStarted","Data":"01bd36261861427d2d19b545ec6e753046dfb299e54d22b57176d4b0470ba952"}
Oct 30 13:37:33.311387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.310912    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:37:33.311387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.310977    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 13:37:33.731490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.729723    4993 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:37:33.731490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:33.729785    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 13:37:34.298658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:34.298584    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" event={"ID":"a72cc892-2538-41a7-8c6a-861a7775e4d7","Type":"ContainerStarted","Data":"664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5"}
Oct 30 13:37:34.313933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:34.313886    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:37:34.314324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:34.314287    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 13:37:34.369591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:34.369525    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podStartSLOduration=3.3695041 podStartE2EDuration="3.3695041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:34.360751408 +0000 UTC m=+3918.836471020" watchObservedRunningTime="2026-10-30 13:37:34.3695041 +0000 UTC m=+3918.845223712"
Oct 30 13:37:34.955843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:34.955796    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:34.964766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:34.964731    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:35.122433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:35.122395    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:37:35.125417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:35.125384    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:37:35.307980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:35.307940    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:35.308479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:35.308312    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:37:35.308838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:35.308816    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:37:35.308928 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:35.308874    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:37:35.309619 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:35.309602    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:37:38.618363 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:37:38.618318    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:37:43.475983 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.475934    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"]
Oct 30 13:37:43.666396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.666357    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"]
Oct 30 13:37:43.676483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.669379    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.787405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.676337    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk" podUID="90039dac-eba4-4da5-854e-ed5e146756ef" containerName="main" probeResult="failure" output="Get \"https://10.132.1.48:8000/ready\": context deadline exceeded"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.816792    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-af8369-openshift-default-dockercfg-nbpm5\""
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.827711    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-socket\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.827813    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403bfbb-dd29-451b-9477-5ffae3262362-istiod-ca-cert\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.827907    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876x7\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-kube-api-access-876x7\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.827947    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-data\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.828019    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-istio-token\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.828166    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-certs\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.828201    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-credential-socket\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.828242    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-envoy\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.879233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.828394    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403bfbb-dd29-451b-9477-5ffae3262362-istio-podinfo\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.934500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.934452    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-socket\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.934789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.934769    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403bfbb-dd29-451b-9477-5ffae3262362-istiod-ca-cert\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.934978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.934957    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876x7\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-kube-api-access-876x7\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.937570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.937535    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-data\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.937824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.937801    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-istio-token\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.948247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.938102    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-certs\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.948614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.948574    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-credential-socket\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.948789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.948772    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-envoy\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.949043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.949028    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403bfbb-dd29-451b-9477-5ffae3262362-istio-podinfo\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.950357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.950331    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403bfbb-dd29-451b-9477-5ffae3262362-istiod-ca-cert\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.964967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.964930    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-certs\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.965825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.965402    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-credential-socket\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.966555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.966532    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-data\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:43.976239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:43.974572    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-socket\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:44.003015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.002933    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403bfbb-dd29-451b-9477-5ffae3262362-istio-podinfo\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:44.003541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.003482    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-envoy\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:44.004250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.004227    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-istio-token\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:44.006031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.006006    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-876x7\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-kube-api-access-876x7\") pod \"e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") " pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:44.171268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.171229    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:44.389137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.388477    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:44.518364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.518311    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:44.795538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.793607    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk" podUID="90039dac-eba4-4da5-854e-ed5e146756ef" containerName="main" probeResult="failure" output="Get \"https://10.132.1.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:44.795538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.794357    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:37:44.863557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.863058    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:37:44.916553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:44.915328    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 30 13:37:45.188637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:45.188586    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"]
Oct 30 13:37:45.198056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:45.197984    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:45.198288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:45.198164    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:45.198381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:45.198293    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:37:46.050102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:46.050042    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" event={"ID":"d403bfbb-dd29-451b-9477-5ffae3262362","Type":"ContainerStarted","Data":"e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281"}
Oct 30 13:37:46.050561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:46.050122    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" event={"ID":"d403bfbb-dd29-451b-9477-5ffae3262362","Type":"ContainerStarted","Data":"66447a1f60cfd479b82f6cf5f5336fc95d7e89401607ce0c25932f56eeef422b"}
Oct 30 13:37:46.183239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:46.176115    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:46.202885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:46.202534    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:46.316290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:46.311118    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podStartSLOduration=3.311094683 podStartE2EDuration="3.311094683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:46.160583105 +0000 UTC m=+3930.636302739" watchObservedRunningTime="2026-10-30 13:37:46.311094683 +0000 UTC m=+3930.786814294"
Oct 30 13:37:47.089559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:47.089515    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:47.097815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:47.097777    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:37:52.056413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:52.056365    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:37:52.057070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:52.056463    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:56.057792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:56.057676    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"]
Oct 30 13:37:56.065003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:56.064944    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4a7799-kgxrh"]
Oct 30 13:37:56.240497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:56.240442    4993 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:37:56.240703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:56.240600    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:57.779410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:37:57.778790    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9009be-b913-42d6-bd52-2a8d3c57dd40" path="/var/lib/kubelet/pods/fe9009be-b913-42d6-bd52-2a8d3c57dd40/volumes"
Oct 30 13:38:00.450155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.450057    4993 generic.go:356] "Generic (PLEG): container finished" podID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerID="d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7" exitCode=0
Oct 30 13:38:00.450799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.450758    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" event={"ID":"ed8cd26d-ec22-4f27-9565-26f6e066c1f8","Type":"ContainerDied","Data":"d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7"}
Oct 30 13:38:00.451550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.451528    4993 scope.go:122] "RemoveContainer" containerID="d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7"
Oct 30 13:38:00.453295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.453188    4993 generic.go:356] "Generic (PLEG): container finished" podID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerID="ac9f5c3d7811277e72cd240634adc79a58747b47ff25350ee37fede2f4a5cca1" exitCode=0
Oct 30 13:38:00.453431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.453408    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" event={"ID":"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3","Type":"ContainerDied","Data":"ac9f5c3d7811277e72cd240634adc79a58747b47ff25350ee37fede2f4a5cca1"}
Oct 30 13:38:00.454657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.453901    4993 scope.go:122] "RemoveContainer" containerID="ac9f5c3d7811277e72cd240634adc79a58747b47ff25350ee37fede2f4a5cca1"
Oct 30 13:38:00.456253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.456216    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:00.456334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.456311    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:00.456375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.456359    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:00.460661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.460629    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:00.460755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.460721    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:00.460814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:00.460770    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:01.494474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:01.494436    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" event={"ID":"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3","Type":"ContainerStarted","Data":"8a7e09385749d2f4f65027bee9a288d5f9371bb9a7c2f19644cc19c38034d3ec"}
Oct 30 13:38:01.497766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:01.497726    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" event={"ID":"ed8cd26d-ec22-4f27-9565-26f6e066c1f8","Type":"ContainerStarted","Data":"071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d"}
Oct 30 13:38:01.610092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:01.610051    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:38:01.610806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:01.610775    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:38:01.610972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:01.610845    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 13:38:01.699477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:01.699356    4993 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 13:38:01.699477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:01.699447    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 13:38:01.705865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:01.705835    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:38:02.614478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:02.614434    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:38:02.615029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:02.614500    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 13:38:02.701229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:02.700696    4993 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 13:38:02.701229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:02.700761    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 13:38:03.611405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:03.611357    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:38:03.611753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:03.611719    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 13:38:03.704855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:03.704806    4993 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 13:38:03.705450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:03.704879    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 13:38:04.610783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:04.610743    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:38:04.611124 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:04.611099    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 13:38:04.698664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:04.698620    4993 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 13:38:04.698982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:04.698951    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 13:38:05.616256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:05.613693    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:38:05.616256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:05.613774    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 13:38:05.722519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:05.721799    4993 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 13:38:05.722519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:05.721858    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 13:38:06.023231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:06.021320    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:38:06.613156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:06.613113    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:38:06.613484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:06.613452    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 13:38:06.706740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:06.706693    4993 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 13:38:06.707387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:06.707332    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 13:38:07.610769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:07.610714    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:38:07.611362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:07.611313    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 13:38:07.718262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:07.718190    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:38:07.719047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:07.719015    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:38:07.719983 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:07.719959    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:38:08.032771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.032725    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"]
Oct 30 13:38:08.037622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.037582    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0edbdcb-l8dr4"]
Oct 30 13:38:08.573632 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:38:08.573592    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:38:08.616067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.616028    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:38:08.617277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.617252    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:38:08.674789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.674652    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"]
Oct 30 13:38:08.690971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.690940    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.708400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.708368    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9a93bd\"/\"default-dockercfg-ktjl4\""
Oct 30 13:38:08.708803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.708512    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9a93bd\"/\"kube-root-ca.crt\""
Oct 30 13:38:08.708803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.708625    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9a93bd\"/\"openshift-service-ca.crt\""
Oct 30 13:38:08.708803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.708772    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9a93bd\"/\"test-model-9a93bd-kserve-self-signed-certs\""
Oct 30 13:38:08.709599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.709570    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"]
Oct 30 13:38:08.809364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.809294    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-dshm\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.809899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.809485    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-model-cache\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.809899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.809539    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tls-certs\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.809899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.809815    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tmp-dir\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.809899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.809893    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-home\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.810148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.810022    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cz2v\" (UniqueName: \"kubernetes.io/projected/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-kube-api-access-8cz2v\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.912422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.912323    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tmp-dir\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.912422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.912413    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-home\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.912629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.912525    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cz2v\" (UniqueName: \"kubernetes.io/projected/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-kube-api-access-8cz2v\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.912629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.912567    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-dshm\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.912709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.912678    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-model-cache\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.912757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.912731    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tls-certs\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.912815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.912804    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tmp-dir\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.913569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.913537    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-home\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.913689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.913616    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-model-cache\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.918002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.917971    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-dshm\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.918813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.918788    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tls-certs\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:08.947598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:08.947563    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cz2v\" (UniqueName: \"kubernetes.io/projected/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-kube-api-access-8cz2v\") pod \"test-model-9a93bd-kserve-55f9488bd7-6n79p\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") " pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:09.019463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:09.018582    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:09.671886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:09.671852    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"]
Oct 30 13:38:09.677388 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:38:09.676439    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb96f62f9_a6d6_459d_a3f3_59ea00bafaac.slice/crio-7443090734ec3ee49b87437e4e1aad64e3f6ee04eb294e2f82d27e859c2793c3 WatchSource:0}: Error finding container 7443090734ec3ee49b87437e4e1aad64e3f6ee04eb294e2f82d27e859c2793c3: Status 404 returned error can't find the container with id 7443090734ec3ee49b87437e4e1aad64e3f6ee04eb294e2f82d27e859c2793c3
Oct 30 13:38:09.694735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:09.694409    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e388c5d0-9415-4eae-9b73-ebc02fcb8e13" path="/var/lib/kubelet/pods/e388c5d0-9415-4eae-9b73-ebc02fcb8e13/volumes"
Oct 30 13:38:09.703978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:09.703940    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p" event={"ID":"b96f62f9-a6d6-459d-a3f3-59ea00bafaac","Type":"ContainerStarted","Data":"7443090734ec3ee49b87437e4e1aad64e3f6ee04eb294e2f82d27e859c2793c3"}
Oct 30 13:38:10.715101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:10.715057    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p" event={"ID":"b96f62f9-a6d6-459d-a3f3-59ea00bafaac","Type":"ContainerStarted","Data":"9fb90b29d4d72f30583dd5abbbc9b395a04f4a93b6845dc3bb7d46effa7bd05a"}
Oct 30 13:38:10.766550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:10.766479    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p" podStartSLOduration=2.7664544319999997 podStartE2EDuration="2.766454432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:10.75776183 +0000 UTC m=+3955.233481441" watchObservedRunningTime="2026-10-30 13:38:10.766454432 +0000 UTC m=+3955.242174043"
Oct 30 13:38:11.020735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:11.020690    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:19.020449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:19.020408    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:19.033504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:19.033476    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:19.898802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:19.898770    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:38:21.471089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:21.471055    4993 scope.go:122] "RemoveContainer" containerID="cc3af79210815a5b001772feff1805af27492b330700913ca5a06400e9fd7b4f"
Oct 30 13:38:29.523966 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:29.523920    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:38:38.605571 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:38:38.605531    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:38:42.036520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:42.036473    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"]
Oct 30 13:38:42.047401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:42.047358    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c966c19d-gbr26"]
Oct 30 13:38:43.755606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:43.755563    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f667967a-7a0c-4e2b-8bd0-2e32980755c9" path="/var/lib/kubelet/pods/f667967a-7a0c-4e2b-8bd0-2e32980755c9/volumes"
Oct 30 13:38:52.039863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:52.039821    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"]
Oct 30 13:38:52.054942 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:52.054904    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f1fe5a-6w4vd"]
Oct 30 13:38:53.758258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:53.758200    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0e335c-5081-4dc0-990d-1797c7cbb766" path="/var/lib/kubelet/pods/5e0e335c-5081-4dc0-990d-1797c7cbb766/volumes"
Oct 30 13:38:59.698227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:59.698164    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"]
Oct 30 13:38:59.698833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:38:59.698495    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy" containerID="cri-o://b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2" gracePeriod=30
Oct 30 13:39:05.014306 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.014272    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:39:05.067271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067234    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-certs\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.067486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067347    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-socket\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.067486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067386    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-data\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.067486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067421    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-envoy\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.067486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067475    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-podinfo\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.067729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067585    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-certs" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.067729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067712    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-socket" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.067842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067729    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-data" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.067842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067759    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-kube-api-access-78tfc\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-kube-api-access-78tfc\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.067958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067865    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-credential-socket\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.067958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067940    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-token\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-token\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.068064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.067995    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istiod-ca-cert\") pod \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\" (UID: \"6f3dfbab-21c7-43e3-a95b-197c84e685c7\") "
Oct 30 13:39:05.068157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.068134    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-credential-socket" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.068570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.068501    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istiod-ca-cert" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:05.068713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.068693    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.068785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.068719    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.068785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.068735    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.068785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.068748    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.068785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.068763    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.071820 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.071791    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-envoy" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.072356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.072336    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-podinfo" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:05.072859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.072814    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-kube-api-access-78tfc" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "kube-api-access-78tfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:05.082465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.082402    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-token" pod "6f3dfbab-21c7-43e3-a95b-197c84e685c7" (UID: "6f3dfbab-21c7-43e3-a95b-197c84e685c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:05.170237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.170187    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.170237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.170233    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.170237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.170243    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3dfbab-21c7-43e3-a95b-197c84e685c7-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.170499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.170253    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78tfc\" (UniqueName: \"kubernetes.io/projected/6f3dfbab-21c7-43e3-a95b-197c84e685c7-kube-api-access-78tfc\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.495296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.495253    4993 generic.go:356] "Generic (PLEG): container finished" podID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerID="b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2" exitCode=0
Oct 30 13:39:05.495296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.495284    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"
Oct 30 13:39:05.495601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.495336    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" event={"ID":"6f3dfbab-21c7-43e3-a95b-197c84e685c7","Type":"ContainerDied","Data":"b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2"}
Oct 30 13:39:05.495601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.495417    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q" event={"ID":"6f3dfbab-21c7-43e3-a95b-197c84e685c7","Type":"ContainerDied","Data":"8cb7fce55148052a1e42b61246f50180a610630edf2d57a3701c14912e2c0f6c"}
Oct 30 13:39:05.495601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.495447    4993 scope.go:122] "RemoveContainer" containerID="b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2"
Oct 30 13:39:05.509734 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.509710    4993 scope.go:122] "RemoveContainer" containerID="b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2"
Oct 30 13:39:05.510121 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:39:05.510093    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2\": container with ID starting with b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2 not found: ID does not exist" containerID="b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2"
Oct 30 13:39:05.510182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.510133    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2"} err="failed to get container status \"b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2\": rpc error: code = NotFound desc = could not find container \"b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2\": container with ID starting with b560120faedb07f1dee6ba390585f4f54689ae01c74327721280bef2718b4da2 not found: ID does not exist"
Oct 30 13:39:05.551944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.551910    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"]
Oct 30 13:39:05.560289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.560251    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-k879q"]
Oct 30 13:39:05.703105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:05.703063    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" path="/var/lib/kubelet/pods/6f3dfbab-21c7-43e3-a95b-197c84e685c7/volumes"
Oct 30 13:39:08.575721 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:39:08.575675    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:39:21.541519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:21.541483    4993 scope.go:122] "RemoveContainer" containerID="765f8d9055680f9f00ecd92a168fcef9f87aabbe3dfe5f2920875f47f4580a23"
Oct 30 13:39:21.553573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:21.553551    4993 scope.go:122] "RemoveContainer" containerID="c82b84cdc81d81301a437cc504e192596906e8e3f6672161f987d3360fe9358b"
Oct 30 13:39:37.041282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:37.041237    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"]
Oct 30 13:39:37.052924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:37.052879    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01bdc7d-6qh2m"]
Oct 30 13:39:37.695526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:37.695480    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52" path="/var/lib/kubelet/pods/d6b9fff7-ddc2-44f7-bf2d-f0bb489d2a52/volumes"
Oct 30 13:39:38.579059 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:39:38.579010    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:39:47.040278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:47.040232    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"]
Oct 30 13:39:47.049630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:47.049595    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58918eab-wb6tq"]
Oct 30 13:39:47.701361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:47.701320    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a086da-85a9-4331-9cbf-f46ab6f3fe86" path="/var/lib/kubelet/pods/d5a086da-85a9-4331-9cbf-f46ab6f3fe86/volumes"
Oct 30 13:39:48.130360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.130318    4993 patch_prober.go:29] interesting pod/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:39:48.130873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.130388    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:39:48.724466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.724422    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"]
Oct 30 13:39:48.725487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.725463    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy"
Oct 30 13:39:48.725626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.725489    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy"
Oct 30 13:39:48.725695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.725673    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3dfbab-21c7-43e3-a95b-197c84e685c7" containerName="istio-proxy"
Oct 30 13:39:48.851603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.851561    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"]
Oct 30 13:39:48.851831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.851690    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:48.879637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.879386    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-025c7f\"/\"kube-root-ca.crt\""
Oct 30 13:39:48.879637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.879528    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-025c7f\"/\"test-model-025c7f-kserve-self-signed-certs\""
Oct 30 13:39:48.879637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.879585    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-025c7f\"/\"default-dockercfg-b27ft\""
Oct 30 13:39:48.879637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.879596    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-025c7f\"/\"openshift-service-ca.crt\""
Oct 30 13:39:49.002291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.999304    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tmp-dir\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.002291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.999373    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-model-cache\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.002291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.999469    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-dshm\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.002291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.999540    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tls-certs\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.002291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.999781    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4z8t\" (UniqueName: \"kubernetes.io/projected/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-kube-api-access-d4z8t\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.002291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:48.999818    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-home\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.105614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.105567    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tls-certs\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.106240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.106191    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4z8t\" (UniqueName: \"kubernetes.io/projected/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-kube-api-access-d4z8t\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.106417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.106400    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-home\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.106618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.106603    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tmp-dir\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.106788 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.106767    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-model-cache\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.107068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.107052    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-dshm\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.108053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.108028    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tmp-dir\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.109085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.108424    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-model-cache\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.110111 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.110077    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-home\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.127792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.127408    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tls-certs\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.153671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.153633    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-dshm\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.283357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.283229    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4z8t\" (UniqueName: \"kubernetes.io/projected/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-kube-api-access-d4z8t\") pod \"test-model-025c7f-kserve-59878bf66f-fxm65\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") " pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:49.562893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:49.562796    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:50.232472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:50.228895    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"]
Oct 30 13:39:50.232472 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:39:50.231753    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb7a5cbe_65aa_40a9_9f8e_ef4b1a19c41b.slice/crio-d2af08effd1873dcdc1fe337b3d287fea5e085ee954d9b412309b5443a0f5330 WatchSource:0}: Error finding container d2af08effd1873dcdc1fe337b3d287fea5e085ee954d9b412309b5443a0f5330: Status 404 returned error can't find the container with id d2af08effd1873dcdc1fe337b3d287fea5e085ee954d9b412309b5443a0f5330
Oct 30 13:39:50.263303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:50.260941    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" event={"ID":"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b","Type":"ContainerStarted","Data":"d2af08effd1873dcdc1fe337b3d287fea5e085ee954d9b412309b5443a0f5330"}
Oct 30 13:39:51.272144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:51.272073    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" event={"ID":"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b","Type":"ContainerStarted","Data":"62160013bd807ef3f6f80c0673e77b55b7dae018bd2e157bcb216373c8d58bba"}
Oct 30 13:39:51.314580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:51.314508    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" podStartSLOduration=3.314488254 podStartE2EDuration="3.314488254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:51.31305997 +0000 UTC m=+4055.788779581" watchObservedRunningTime="2026-10-30 13:39:51.314488254 +0000 UTC m=+4055.790207866"
Oct 30 13:39:51.349695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:51.349323    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:39:51.349695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:51.349402    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:39:51.563369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:51.563276    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:58.364377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:58.364324    4993 generic.go:356] "Generic (PLEG): container finished" podID="d403bfbb-dd29-451b-9477-5ffae3262362" containerID="e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281" exitCode=0
Oct 30 13:39:58.364907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:58.364644    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" event={"ID":"d403bfbb-dd29-451b-9477-5ffae3262362","Type":"ContainerDied","Data":"e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281"}
Oct 30 13:39:58.365379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:58.365351    4993 scope.go:122] "RemoveContainer" containerID="e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281"
Oct 30 13:39:58.367879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:58.367843    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:39:58.368001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:58.367950    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:39:58.368075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:58.368006    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:39:59.379136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:59.379094    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" event={"ID":"d403bfbb-dd29-451b-9477-5ffae3262362","Type":"ContainerStarted","Data":"8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca"}
Oct 30 13:39:59.564649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:59.564607    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:39:59.600677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:39:59.600643    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:40:00.174363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:00.173379    4993 patch_prober.go:29] interesting pod/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body=
Oct 30 13:40:00.174363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:00.173443    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused"
Oct 30 13:40:00.174363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:00.173737    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:40:00.425506 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:00.425416    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:40:01.095811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.095718    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"]
Oct 30 13:40:01.105096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.105045    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dad88a8-st7cf"]
Oct 30 13:40:01.176660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.176618    4993 patch_prober.go:29] interesting pod/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body=
Oct 30 13:40:01.177407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.177367    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused"
Oct 30 13:40:01.189977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.189926    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"]
Oct 30 13:40:01.222950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.221856    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.235770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.231845    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"]
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.252696    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s874c\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-kube-api-access-s874c\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.252783    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.252835    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.252938    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.253000    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2710cdd-82a3-424e-938b-372e520fc95c-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.253107    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.253164    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.253228    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.271231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.253255    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2710cdd-82a3-424e-938b-372e520fc95c-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.375292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.372849    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.375292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.372934    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.375292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.372978    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.375292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.373007    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2710cdd-82a3-424e-938b-372e520fc95c-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.375292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.373138    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s874c\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-kube-api-access-s874c\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.375292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.373200    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.376230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.375906    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.376230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.376164    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.386232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.376431    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.393900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.393508    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.393900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.393674    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.393900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.393755    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2710cdd-82a3-424e-938b-372e520fc95c-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.394895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.394695    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2710cdd-82a3-424e-938b-372e520fc95c-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.400708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.396300    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.400708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.398480    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.405937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.405899    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2710cdd-82a3-424e-938b-372e520fc95c-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.410411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.410159    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.410411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.410382    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s874c\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-kube-api-access-s874c\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.577364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.577318    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:01.699074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:01.698962    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781323a9-1d50-4762-8026-4c2268b7327b" path="/var/lib/kubelet/pods/781323a9-1d50-4762-8026-4c2268b7327b/volumes"
Oct 30 13:40:02.108184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.108137    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:40:02.191275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.188896    4993 patch_prober.go:29] interesting pod/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body=
Oct 30 13:40:02.191275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.188982    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused"
Oct 30 13:40:02.211354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.211320    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"]
Oct 30 13:40:02.215850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.215813    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:02.216028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.216003    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:02.216087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.216061    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:02.430374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.429276    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" event={"ID":"e2710cdd-82a3-424e-938b-372e520fc95c","Type":"ContainerStarted","Data":"5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2"}
Oct 30 13:40:02.430374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.429345    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" event={"ID":"e2710cdd-82a3-424e-938b-372e520fc95c","Type":"ContainerStarted","Data":"8380c61eb45aa3131b08c6fc34eaa02fd7e7785c710a0ef0b7d250103628ddaf"}
Oct 30 13:40:02.502227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.502135    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" podStartSLOduration=1.5021193099999999 podStartE2EDuration="1.50211931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:02.497599829 +0000 UTC m=+4066.973319441" watchObservedRunningTime="2026-10-30 13:40:02.50211931 +0000 UTC m=+4066.977838917"
Oct 30 13:40:02.583314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:02.581453    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:03.177240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.176652    4993 patch_prober.go:29] interesting pod/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body=
Oct 30 13:40:03.177240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.176720    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused"
Oct 30 13:40:03.439709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.439626    4993 generic.go:356] "Generic (PLEG): container finished" podID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerID="664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5" exitCode=0
Oct 30 13:40:03.439965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.439699    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" event={"ID":"a72cc892-2538-41a7-8c6a-861a7775e4d7","Type":"ContainerDied","Data":"664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5"}
Oct 30 13:40:03.440521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.440497    4993 scope.go:122] "RemoveContainer" containerID="664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5"
Oct 30 13:40:03.442562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.442529    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:03.442668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.442642    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:03.442736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.442695    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:03.618253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.617365    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 13:40:03.618253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:03.617433    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 13:40:04.175230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:04.173639    4993 patch_prober.go:29] interesting pod/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body=
Oct 30 13:40:04.175230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:04.173720    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused"
Oct 30 13:40:04.468332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:04.468233    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" event={"ID":"a72cc892-2538-41a7-8c6a-861a7775e4d7","Type":"ContainerStarted","Data":"66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00"}
Oct 30 13:40:04.579234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:04.578717    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 13:40:04.579234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:04.578789    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 13:40:04.973229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:04.959179    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:40:04.973229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:04.959387    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:40:04.973229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:04.959456    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 30 13:40:05.040394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.040354    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"]
Oct 30 13:40:05.044706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.044676    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b68733d-wfs72"]
Oct 30 13:40:05.174233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.172619    4993 patch_prober.go:29] interesting pod/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body=
Oct 30 13:40:05.174233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.172682    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused"
Oct 30 13:40:05.320230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.310146    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:40:05.583298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.582662    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 13:40:05.583298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.582727    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 13:40:05.735009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.734962    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3089258-552e-40b5-82a0-d4b7a96e5b58" path="/var/lib/kubelet/pods/e3089258-552e-40b5-82a0-d4b7a96e5b58/volumes"
Oct 30 13:40:05.962299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.960677    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:40:05.962299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:05.960747    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 30 13:40:06.201232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:06.200532    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:40:06.211233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:06.206424    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:40:06.607239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:06.588113    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 13:40:06.618008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:06.617337    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 13:40:06.973290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:06.967567    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:40:06.973290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:06.967630    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 30 13:40:07.584230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:07.583760    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 13:40:07.584230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:07.583842    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 13:40:07.973291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:07.956562    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:40:07.973291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:07.956626    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 30 13:40:08.587226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:08.586606    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body=
Oct 30 13:40:08.587226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:08.586676    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused"
Oct 30 13:40:08.737269 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:40:08.737161    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:40:08.956660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:08.956570    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:40:08.956935 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:08.956901    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 30 13:40:09.591017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:09.590981    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:09.591661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:09.591632    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:09.592595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:09.592572    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:40:09.956735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:09.956637    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:40:09.960336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:09.960284    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 30 13:40:10.956668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:10.956618    4993 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:40:10.957257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:10.956692    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 30 13:40:11.034680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:11.034643    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"]
Oct 30 13:40:11.042768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:11.042735    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd907d5e-t88sg"]
Oct 30 13:40:11.697198 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:11.697160    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7302120-3f5a-4fdf-ae46-685d1394dd41" path="/var/lib/kubelet/pods/b7302120-3f5a-4fdf-ae46-685d1394dd41/volumes"
Oct 30 13:40:11.963039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:11.962962    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:40:11.964037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:11.964015    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:40:15.725684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.725645    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"]
Oct 30 13:40:15.738553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.738507    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.755924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.755890    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"]
Oct 30 13:40:15.758743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.758714    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f97234\"/\"openshift-service-ca.crt\""
Oct 30 13:40:15.758743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.758737    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f97234\"/\"test-model-f97234-kserve-self-signed-certs\""
Oct 30 13:40:15.758962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.758838    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f97234\"/\"kube-root-ca.crt\""
Oct 30 13:40:15.758962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.758917    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f97234\"/\"default-dockercfg-zbr4m\""
Oct 30 13:40:15.824975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.824931    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-model-cache\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.825199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.825018    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt96r\" (UniqueName: \"kubernetes.io/projected/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-kube-api-access-gt96r\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.825565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.825536    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tls-certs\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.825883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.825856    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-dshm\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.826093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.826069    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-home\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.826348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.826324    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tmp-dir\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.929021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.928970    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-model-cache\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.929363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.929342    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt96r\" (UniqueName: \"kubernetes.io/projected/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-kube-api-access-gt96r\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.929589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.929574    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tls-certs\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.929744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.929729    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-dshm\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.929876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.929863    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-home\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.930061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.930047    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tmp-dir\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.931180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.931150    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-model-cache\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.931762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.931678    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tmp-dir\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.931762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.931714    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-home\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.933960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.933932    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-dshm\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.935164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.935137    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tls-certs\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:15.960906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:15.960865    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt96r\" (UniqueName: \"kubernetes.io/projected/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-kube-api-access-gt96r\") pod \"test-model-f97234-kserve-5969dd6cc-2pvsm\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") " pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:16.092243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:16.083264    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"]
Oct 30 13:40:16.100250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:16.094291    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f97234\"/\"default-dockercfg-zbr4m\""
Oct 30 13:40:16.106241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:16.104326    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:16.134235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:16.133425    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfbe4e1-8mr84"]
Oct 30 13:40:16.761177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:16.760647    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"]
Oct 30 13:40:16.765373 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:40:16.765337    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f4a305e_72ba_4da6_bb86_d0ac5dfbb8a7.slice/crio-702857cc736e12e50171de8dea6f0d180081b40c8c1a93223764b2b5a8146202 WatchSource:0}: Error finding container 702857cc736e12e50171de8dea6f0d180081b40c8c1a93223764b2b5a8146202: Status 404 returned error can't find the container with id 702857cc736e12e50171de8dea6f0d180081b40c8c1a93223764b2b5a8146202
Oct 30 13:40:16.904562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:16.904485    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" event={"ID":"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7","Type":"ContainerStarted","Data":"4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403"}
Oct 30 13:40:16.904562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:16.904540    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" event={"ID":"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7","Type":"ContainerStarted","Data":"702857cc736e12e50171de8dea6f0d180081b40c8c1a93223764b2b5a8146202"}
Oct 30 13:40:16.950695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:16.950625    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" podStartSLOduration=1.950608629 podStartE2EDuration="1.950608629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:16.946141814 +0000 UTC m=+4081.421861427" watchObservedRunningTime="2026-10-30 13:40:16.950608629 +0000 UTC m=+4081.426328239"
Oct 30 13:40:17.108279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:17.107593    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:17.703694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:17.703647    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c06df3-073d-49ab-bd94-f4c9e41e6f56" path="/var/lib/kubelet/pods/f9c06df3-073d-49ab-bd94-f4c9e41e6f56/volumes"
Oct 30 13:40:21.599007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:21.598966    4993 scope.go:122] "RemoveContainer" containerID="b2ec60823bd57d1eab766fcbbc5779f35aba051f1c76537c301e17f1d9acbe69"
Oct 30 13:40:21.614173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:21.614141    4993 scope.go:122] "RemoveContainer" containerID="d54736adb4ba0ce5136b7ccc1624a935651ed9750eb472c8e450bbb5d7ba4887"
Oct 30 13:40:22.116436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.116396    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"]
Oct 30 13:40:22.153588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.153526    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"]
Oct 30 13:40:22.153856 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.153641    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.188957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.188923    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09ad1e\"/\"default-dockercfg-8lwx4\""
Oct 30 13:40:22.189324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.189303    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09ad1e\"/\"kube-root-ca.crt\""
Oct 30 13:40:22.189645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.189601    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09ad1e\"/\"test-model-09ad1e-kserve-self-signed-certs\""
Oct 30 13:40:22.189807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.189791    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09ad1e\"/\"openshift-service-ca.crt\""
Oct 30 13:40:22.329151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.329108    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-model-cache\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.329456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.329426    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-home\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.329616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.329591    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77pjq\" (UniqueName: \"kubernetes.io/projected/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-kube-api-access-77pjq\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.329685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.329655    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tmp-dir\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.329744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.329702    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tls-certs\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.329862 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.329842    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-dshm\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.434283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.433583    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77pjq\" (UniqueName: \"kubernetes.io/projected/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-kube-api-access-77pjq\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.434283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.433663    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tmp-dir\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.434283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.433716    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tls-certs\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.434283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.433844    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-dshm\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.434283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.434076    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-model-cache\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.434283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.434117    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-home\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.434753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.434650    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-home\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.449232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.435235    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tmp-dir\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.455229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.455101    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-model-cache\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.477238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.467959    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77pjq\" (UniqueName: \"kubernetes.io/projected/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-kube-api-access-77pjq\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.477238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.468565    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tls-certs\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.487305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.477509    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-dshm\") pod \"test-model-09ad1e-kserve-658d8b4df9-kn6pk\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") " pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:22.487305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:22.481338    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:23.042834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:23.042795    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"]
Oct 30 13:40:23.055808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:23.055765    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk" event={"ID":"5bf8bf0b-166c-44c4-be3e-0b8488950bb3","Type":"ContainerStarted","Data":"2bc63b75e509fa0a02b0aab552e6d011ed84bdd7cef0b3460838d6f26af0ce3e"}
Oct 30 13:40:24.067006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:24.066959    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk" event={"ID":"5bf8bf0b-166c-44c4-be3e-0b8488950bb3","Type":"ContainerStarted","Data":"bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e"}
Oct 30 13:40:24.116433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:24.116364    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk" podStartSLOduration=2.116349532 podStartE2EDuration="2.116349532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:24.114142619 +0000 UTC m=+4088.589862220" watchObservedRunningTime="2026-10-30 13:40:24.116349532 +0000 UTC m=+4088.592069187"
Oct 30 13:40:24.481844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:24.481811    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:26.105787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:26.105691    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:26.119422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:26.119383    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:27.109504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:27.109472    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:40:32.482038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:32.481982    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:32.503481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:32.503447    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:33.218425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:33.218393    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:40:38.575895 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:40:38.575851    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:40:46.368830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.368790    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"]
Oct 30 13:40:46.374488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.374467    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.402517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.402433    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:40:46.402777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.402704    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:40:46.402884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.402798    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:40:46.403104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.403043    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:40:46.410051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.410013    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"]
Oct 30 13:40:46.527275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.527104    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.527516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.527447    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.527613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.527528    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpdf\" (UniqueName: \"kubernetes.io/projected/037c4482-a6f7-450d-98a1-23300038fc4a-kube-api-access-qlpdf\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.629076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.628967    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.629076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.629051    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpdf\" (UniqueName: \"kubernetes.io/projected/037c4482-a6f7-450d-98a1-23300038fc4a-kube-api-access-qlpdf\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.629403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.629239    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.629921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.629899    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.631501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.631477    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.656917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.656886    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpdf\" (UniqueName: \"kubernetes.io/projected/037c4482-a6f7-450d-98a1-23300038fc4a-kube-api-access-qlpdf\") pod \"maas-api-revoke-sub-9d3c88b5-jj6hd\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.690412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.690373    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:46.949093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:46.948998    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"]
Oct 30 13:40:47.015251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:47.015197    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"]
Oct 30 13:40:47.023968 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:40:47.023933    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037c4482_a6f7_450d_98a1_23300038fc4a.slice/crio-0753180f61737a89acf27120cbc6bf7fdfee7ea07cfad728175f7b20479aef09 WatchSource:0}: Error finding container 0753180f61737a89acf27120cbc6bf7fdfee7ea07cfad728175f7b20479aef09: Status 404 returned error can't find the container with id 0753180f61737a89acf27120cbc6bf7fdfee7ea07cfad728175f7b20479aef09
Oct 30 13:40:47.366951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:47.366907    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd" event={"ID":"037c4482-a6f7-450d-98a1-23300038fc4a","Type":"ContainerStarted","Data":"0753180f61737a89acf27120cbc6bf7fdfee7ea07cfad728175f7b20479aef09"}
Oct 30 13:40:47.367292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:47.367236    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy" containerID="cri-o://6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2" gracePeriod=30
Oct 30 13:40:48.384144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:48.384106    4993 generic.go:356] "Generic (PLEG): container finished" podID="037c4482-a6f7-450d-98a1-23300038fc4a" containerID="cc8dc70876c2e3b8056e631c39e5ed6fbb5acdb78507775c9db23cde96ae4455" exitCode=0
Oct 30 13:40:48.384597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:48.384262    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd" event={"ID":"037c4482-a6f7-450d-98a1-23300038fc4a","Type":"ContainerDied","Data":"cc8dc70876c2e3b8056e631c39e5ed6fbb5acdb78507775c9db23cde96ae4455"}
Oct 30 13:40:49.597490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.597457    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:49.689675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.689576    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037c4482-a6f7-450d-98a1-23300038fc4a-kube-api-access-qlpdf\" (UniqueName: \"kubernetes.io/projected/037c4482-a6f7-450d-98a1-23300038fc4a-kube-api-access-qlpdf\") pod \"037c4482-a6f7-450d-98a1-23300038fc4a\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") "
Oct 30 13:40:49.689864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.689770    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-openshift-ca\") pod \"037c4482-a6f7-450d-98a1-23300038fc4a\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") "
Oct 30 13:40:49.689938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.689860    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-xks-ca\") pod \"037c4482-a6f7-450d-98a1-23300038fc4a\" (UID: \"037c4482-a6f7-450d-98a1-23300038fc4a\") "
Oct 30 13:40:49.690767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.690737    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-openshift-ca" pod "037c4482-a6f7-450d-98a1-23300038fc4a" (UID: "037c4482-a6f7-450d-98a1-23300038fc4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:49.692476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.692445    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037c4482-a6f7-450d-98a1-23300038fc4a-kube-api-access-qlpdf" pod "037c4482-a6f7-450d-98a1-23300038fc4a" (UID: "037c4482-a6f7-450d-98a1-23300038fc4a"). InnerVolumeSpecName "kube-api-access-qlpdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:49.693645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.693617    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-xks-ca" pod "037c4482-a6f7-450d-98a1-23300038fc4a" (UID: "037c4482-a6f7-450d-98a1-23300038fc4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:49.791974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.791794    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlpdf\" (UniqueName: \"kubernetes.io/projected/037c4482-a6f7-450d-98a1-23300038fc4a-kube-api-access-qlpdf\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:49.791974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.791904    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:49.791974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:49.791951    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037c4482-a6f7-450d-98a1-23300038fc4a-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.403969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:50.403935    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"
Oct 30 13:40:50.404172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:50.403933    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd" event={"ID":"037c4482-a6f7-450d-98a1-23300038fc4a","Type":"ContainerDied","Data":"0753180f61737a89acf27120cbc6bf7fdfee7ea07cfad728175f7b20479aef09"}
Oct 30 13:40:50.404172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:50.404082    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0753180f61737a89acf27120cbc6bf7fdfee7ea07cfad728175f7b20479aef09"
Oct 30 13:40:52.682584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.682542    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 13:40:52.749709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.749675    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-socket\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.749893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.749774    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-podinfo\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.749893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.749797    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-certs\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.750019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.749919    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-kube-api-access-t5zb5\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-kube-api-access-t5zb5\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.750080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750020    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istiod-ca-cert\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.750080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750032    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-socket" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.750200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750105    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-data\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.750200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750152    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-token\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-token\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.750200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750197    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-certs" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.750477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750265    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-credential-socket\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.750477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750335    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-envoy\") pod \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\" (UID: \"f9d46308-8d87-4f1f-b898-0d1dfbba8f03\") "
Oct 30 13:40:52.750631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750485    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-data" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.750818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.750796    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istiod-ca-cert" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:52.751137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.751115    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-credential-socket" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.751362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.751175    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.751497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.751375    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.751497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.751410    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.751497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.751442    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.760256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.754897    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-kube-api-access-t5zb5" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "kube-api-access-t5zb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:52.760256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.754939    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-token" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:52.760256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.755020    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-podinfo" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:52.760256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.755644    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-envoy" pod "f9d46308-8d87-4f1f-b898-0d1dfbba8f03" (UID: "f9d46308-8d87-4f1f-b898-0d1dfbba8f03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.852927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.852878    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.852927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.852917    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.852927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.852927    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.852927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.852936    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.853407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:52.852944    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zb5\" (UniqueName: \"kubernetes.io/projected/f9d46308-8d87-4f1f-b898-0d1dfbba8f03-kube-api-access-t5zb5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.429692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.429657    4993 generic.go:356] "Generic (PLEG): container finished" podID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerID="6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2" exitCode=0
Oct 30 13:40:53.429880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.429707    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"
Oct 30 13:40:53.429880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.429738    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" event={"ID":"f9d46308-8d87-4f1f-b898-0d1dfbba8f03","Type":"ContainerDied","Data":"6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2"}
Oct 30 13:40:53.429880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.429792    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9" event={"ID":"f9d46308-8d87-4f1f-b898-0d1dfbba8f03","Type":"ContainerDied","Data":"62e87be22807352241d5487254ef77fbe23666b464442fcc6a17d8891ce19032"}
Oct 30 13:40:53.429880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.429816    4993 scope.go:122] "RemoveContainer" containerID="6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2"
Oct 30 13:40:53.442636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.442612    4993 scope.go:122] "RemoveContainer" containerID="6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2"
Oct 30 13:40:53.442967 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:40:53.442946    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2\": container with ID starting with 6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2 not found: ID does not exist" containerID="6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2"
Oct 30 13:40:53.443056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.442979    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2"} err="failed to get container status \"6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2\": rpc error: code = NotFound desc = could not find container \"6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2\": container with ID starting with 6fe3c7f03160a0f570a97d2a9ac9f65025955bbbc30baccfbd41c99c4c183ce2 not found: ID does not exist"
Oct 30 13:40:53.501118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.501079    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"]
Oct 30 13:40:53.506357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.506319    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-dwzh9"]
Oct 30 13:40:53.695600 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:53.695501    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" path="/var/lib/kubelet/pods/f9d46308-8d87-4f1f-b898-0d1dfbba8f03/volumes"
Oct 30 13:40:57.973031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.972989    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"]
Oct 30 13:40:57.973708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.973693    4993 cpu_manager.go:414] "containerMap: removing container" podUID="037c4482-a6f7-450d-98a1-23300038fc4a" containerName="revoke-keys"
Oct 30 13:40:57.973783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.973710    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037c4482-a6f7-450d-98a1-23300038fc4a" containerName="revoke-keys"
Oct 30 13:40:57.973783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.973738    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy"
Oct 30 13:40:57.973783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.973744    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy"
Oct 30 13:40:57.973895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.973866    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="037c4482-a6f7-450d-98a1-23300038fc4a" containerName="revoke-keys"
Oct 30 13:40:57.973895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.973880    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d46308-8d87-4f1f-b898-0d1dfbba8f03" containerName="istio-proxy"
Oct 30 13:40:57.979035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.979009    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:57.994222 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.994178    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:40:57.994437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:57.994418    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:40:58.006111 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.006078    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:40:58.006395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.006078    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:40:58.012391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.012354    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"]
Oct 30 13:40:58.102605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.102571    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.102909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.102882    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k9qc\" (UniqueName: \"kubernetes.io/projected/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-kube-api-access-4k9qc\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.103067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.103051    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.204671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.204625    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.204854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.204705    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.204968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.204948    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9qc\" (UniqueName: \"kubernetes.io/projected/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-kube-api-access-4k9qc\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.205474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.205451    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.207344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.207317    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.228065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.228033    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9qc\" (UniqueName: \"kubernetes.io/projected/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-kube-api-access-4k9qc\") pod \"maas-api-revoke-sub-3783efb0-n6xbb\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.296245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.295283    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:40:58.610568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:58.610537    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"]
Oct 30 13:40:59.382305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:59.382195    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"]
Oct 30 13:40:59.382793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:59.382462    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" containerID="cri-o://ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa" gracePeriod=30
Oct 30 13:40:59.631223 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:59.631176    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb" event={"ID":"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d","Type":"ContainerStarted","Data":"435f58a114f2c753c989b717e4a26223d21b214dedbe11890c903ae8a30c4527"}
Oct 30 13:40:59.631425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:59.631287    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb" event={"ID":"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d","Type":"ContainerStarted","Data":"0213e7476a0cc476b9ceec970cabd158d5ddda099a638b647e19463a997ff87c"}
Oct 30 13:40:59.671780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:40:59.671598    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb" podStartSLOduration=2.671584095 podStartE2EDuration="2.671584095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:59.670650018 +0000 UTC m=+4124.146369628" watchObservedRunningTime="2026-10-30 13:40:59.671584095 +0000 UTC m=+4124.147303704"
Oct 30 13:41:00.728376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:00.728333    4993 generic.go:356] "Generic (PLEG): container finished" podID="7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d" containerID="435f58a114f2c753c989b717e4a26223d21b214dedbe11890c903ae8a30c4527" exitCode=0
Oct 30 13:41:00.728904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:00.728519    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb" event={"ID":"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d","Type":"ContainerDied","Data":"435f58a114f2c753c989b717e4a26223d21b214dedbe11890c903ae8a30c4527"}
Oct 30 13:41:01.930798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:01.930765    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:41:01.981869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:01.981825    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-xks-ca\") pod \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") "
Oct 30 13:41:01.982073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:01.982011    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-kube-api-access-4k9qc\" (UniqueName: \"kubernetes.io/projected/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-kube-api-access-4k9qc\") pod \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") "
Oct 30 13:41:01.982146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:01.982115    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-openshift-ca\") pod \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\" (UID: \"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d\") "
Oct 30 13:41:01.982470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:01.982443    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-openshift-ca" pod "7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d" (UID: "7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:01.982788 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:01.982771    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:01.984700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:01.984668    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-xks-ca" pod "7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d" (UID: "7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:01.985121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:01.985095    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-kube-api-access-4k9qc" pod "7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d" (UID: "7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d"). InnerVolumeSpecName "kube-api-access-4k9qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:02.084712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:02.084669    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:02.084712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:02.084713    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k9qc\" (UniqueName: \"kubernetes.io/projected/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d-kube-api-access-4k9qc\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:02.745974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:02.745944    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"
Oct 30 13:41:02.746162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:02.745985    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb" event={"ID":"7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d","Type":"ContainerDied","Data":"0213e7476a0cc476b9ceec970cabd158d5ddda099a638b647e19463a997ff87c"}
Oct 30 13:41:02.746162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:02.746048    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0213e7476a0cc476b9ceec970cabd158d5ddda099a638b647e19463a997ff87c"
Oct 30 13:41:03.604819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:03.604778    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:41:03.605340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:03.604856    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:41:04.715300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.715266    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:41:04.756021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.755986    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-kube-api-access-xwb9g\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-kube-api-access-xwb9g\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.756241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756083    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-credential-socket\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.756241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756156    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-certs\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.756348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756283    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-envoy\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.756425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756403    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-data\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.756510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756491    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-istio-token\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-istio-token\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.756609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756590    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-socket\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.756683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756643    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6349107-300d-4744-8712-9a0712a45e7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6349107-300d-4744-8712-9a0712a45e7e-istio-podinfo\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.756748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756704    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6349107-300d-4744-8712-9a0712a45e7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6349107-300d-4744-8712-9a0712a45e7e-istiod-ca-cert\") pod \"e6349107-300d-4744-8712-9a0712a45e7e\" (UID: \"e6349107-300d-4744-8712-9a0712a45e7e\") "
Oct 30 13:41:04.757310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756851    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-certs" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:04.757310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.756877    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-credential-socket" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:04.757519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.757312    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6349107-300d-4744-8712-9a0712a45e7e-istiod-ca-cert" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:04.757589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.757554    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6349107-300d-4744-8712-9a0712a45e7e-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.757589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.757576    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.757706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.757590    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.757800 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.757691    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-socket" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:04.759259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.759039    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-kube-api-access-xwb9g" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "kube-api-access-xwb9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:04.759653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.759617    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-data" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:04.760170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.760146    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-envoy" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:04.763475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.762343    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6349107-300d-4744-8712-9a0712a45e7e-istio-podinfo" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:04.763475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.763191    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-istio-token" pod "e6349107-300d-4744-8712-9a0712a45e7e" (UID: "e6349107-300d-4744-8712-9a0712a45e7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:04.768239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.768193    4993 generic.go:356] "Generic (PLEG): container finished" podID="e6349107-300d-4744-8712-9a0712a45e7e" containerID="ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa" exitCode=0
Oct 30 13:41:04.768398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.768355    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"
Oct 30 13:41:04.768470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.768415    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" event={"ID":"e6349107-300d-4744-8712-9a0712a45e7e","Type":"ContainerDied","Data":"ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa"}
Oct 30 13:41:04.768470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.768457    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb" event={"ID":"e6349107-300d-4744-8712-9a0712a45e7e","Type":"ContainerDied","Data":"40a066d6159e350ed8e870375e1cae9bd05c89f4ae8ac47d8c4d600afa44fdf3"}
Oct 30 13:41:04.768592 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.768481    4993 scope.go:122] "RemoveContainer" containerID="ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa"
Oct 30 13:41:04.809511 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.809471    4993 scope.go:122] "RemoveContainer" containerID="ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa"
Oct 30 13:41:04.811649 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:41:04.811607    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa\": container with ID starting with ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa not found: ID does not exist" containerID="ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa"
Oct 30 13:41:04.811843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.811660    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa"} err="failed to get container status \"ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa\": rpc error: code = NotFound desc = could not find container \"ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa\": container with ID starting with ca84e89106eedf30711546a78e39304dacea34f85da5472d1ed3ad7c7d48dbaa not found: ID does not exist"
Oct 30 13:41:04.837570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.837522    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"]
Oct 30 13:41:04.842654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.842614    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-gq9mb"]
Oct 30 13:41:04.859759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.859716    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.859759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.859756    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6349107-300d-4744-8712-9a0712a45e7e-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.860061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.859773    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwb9g\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-kube-api-access-xwb9g\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.860061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.859789    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.860061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.859805    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6349107-300d-4744-8712-9a0712a45e7e-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.860061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:04.859817    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6349107-300d-4744-8712-9a0712a45e7e-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:05.698253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:05.698190    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6349107-300d-4744-8712-9a0712a45e7e" path="/var/lib/kubelet/pods/e6349107-300d-4744-8712-9a0712a45e7e/volumes"
Oct 30 13:41:08.577917 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:41:08.577875    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:41:15.205293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:15.184640    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"]
Oct 30 13:41:15.205293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:15.184940    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy" containerID="cri-o://04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06" gracePeriod=30
Oct 30 13:41:20.618734 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.618689    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:41:20.818992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.818895    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-podinfo\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.819288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.819261    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-token\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-token\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.829033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.827126    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-podinfo" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:20.829033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.828163    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-envoy\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.829033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.828272    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-data\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.829033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.828347    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-socket\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.829033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.828422    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-kube-api-access-ks84l\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-kube-api-access-ks84l\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.829033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.828473    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-certs\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.829033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.828713    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb718195-5572-418f-a6be-e3fa3a0fccad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb718195-5572-418f-a6be-e3fa3a0fccad-istiod-ca-cert\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.829033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.828793    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-credential-socket\") pod \"bb718195-5572-418f-a6be-e3fa3a0fccad\" (UID: \"bb718195-5572-418f-a6be-e3fa3a0fccad\") "
Oct 30 13:41:20.830756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.830436    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-socket" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.830756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.830685    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-data" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.832014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.831343    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb718195-5572-418f-a6be-e3fa3a0fccad-istiod-ca-cert" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:20.832014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.831720    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-certs" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.836129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.836081    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-credential-socket" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.849500 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.849469    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"]
Oct 30 13:41:20.853868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.852477    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.853868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.852826    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.853868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.853062    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.853868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.853082    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb718195-5572-418f-a6be-e3fa3a0fccad-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.853868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.853096    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.853868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.853109    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.881238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.876432    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-envoy" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.881238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.876470    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-kube-api-access-ks84l" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "kube-api-access-ks84l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:20.881238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.876560    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-token" pod "bb718195-5572-418f-a6be-e3fa3a0fccad" (UID: "bb718195-5572-418f-a6be-e3fa3a0fccad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:20.980243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.979861    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks84l\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-kube-api-access-ks84l\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.980243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.979907    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.980243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:20.979922    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb718195-5572-418f-a6be-e3fa3a0fccad-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.179320    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"]
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180285    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d" containerName="revoke-keys"
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180303    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d" containerName="revoke-keys"
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180336    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy"
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180354    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy"
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180364    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy"
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180369    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy"
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180545    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d" containerName="revoke-keys"
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180558    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6349107-300d-4744-8712-9a0712a45e7e" containerName="istio-proxy"
Oct 30 13:41:21.183320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.180568    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerName="istio-proxy"
Oct 30 13:41:21.196243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.186808    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.200231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.198323    4993 generic.go:356] "Generic (PLEG): container finished" podID="bb718195-5572-418f-a6be-e3fa3a0fccad" containerID="04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06" exitCode=0
Oct 30 13:41:21.200231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.198580    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" event={"ID":"bb718195-5572-418f-a6be-e3fa3a0fccad","Type":"ContainerDied","Data":"04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06"}
Oct 30 13:41:21.200231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.198675    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn" event={"ID":"bb718195-5572-418f-a6be-e3fa3a0fccad","Type":"ContainerDied","Data":"de3b3242bc1526615979304b17322a117e93175909c7952eba4f5bf0f99e675a"}
Oct 30 13:41:21.200231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.198726    4993 scope.go:122] "RemoveContainer" containerID="04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06"
Oct 30 13:41:21.221175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.200647    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"
Oct 30 13:41:21.221175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.201092    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"]
Oct 30 13:41:21.221175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.203355    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:41:21.221175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.203536    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:41:21.221175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.212473    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:41:21.221175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.212592    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:41:21.260014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.259979    4993 scope.go:122] "RemoveContainer" containerID="04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06"
Oct 30 13:41:21.260478 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:41:21.260449    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06\": container with ID starting with 04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06 not found: ID does not exist" containerID="04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06"
Oct 30 13:41:21.260616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.260594    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06"} err="failed to get container status \"04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06\": rpc error: code = NotFound desc = could not find container \"04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06\": container with ID starting with 04b8534f30a15e8384cacd24ea66180d1800ac99f5138a8a9c20c6e061d05b06 not found: ID does not exist"
Oct 30 13:41:21.291071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.291032    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"]
Oct 30 13:41:21.296737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.296699    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9c7\" (UniqueName: \"kubernetes.io/projected/22356426-47bd-40fb-b64a-df78bf31f32d-kube-api-access-hx9c7\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.297194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.297172    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.297736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.297711    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.304014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.303959    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-28wtn"]
Oct 30 13:41:21.400123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.400075    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.400361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.400190    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9c7\" (UniqueName: \"kubernetes.io/projected/22356426-47bd-40fb-b64a-df78bf31f32d-kube-api-access-hx9c7\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.400361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.400289    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.400998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.400963    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.402940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.402913    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.424345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.424311    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9c7\" (UniqueName: \"kubernetes.io/projected/22356426-47bd-40fb-b64a-df78bf31f32d-kube-api-access-hx9c7\") pod \"maas-api-revoke-keys-e2e-inf-a-d41320-7757z\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.555377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.555336    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:21.657791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.657737    4993 scope.go:122] "RemoveContainer" containerID="648a7e47261ab2d24b213aa443dd4024a71c36333e9aab694518b40e5e94da5c"
Oct 30 13:41:21.679672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.679549    4993 scope.go:122] "RemoveContainer" containerID="a5ed7097fffa465a3d54f2b0077aaaabf788633435c30256d4102b907d0ef20b"
Oct 30 13:41:21.698873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.698803    4993 scope.go:122] "RemoveContainer" containerID="a27aa6a99a6d61f27ab037b3f7cf798ca5ddd9ce996c54ce6bb3772a20802b40"
Oct 30 13:41:21.702416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.702379    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb718195-5572-418f-a6be-e3fa3a0fccad" path="/var/lib/kubelet/pods/bb718195-5572-418f-a6be-e3fa3a0fccad/volumes"
Oct 30 13:41:21.716590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.716558    4993 scope.go:122] "RemoveContainer" containerID="18a4016a6f4e6b527c8807111c6a9926df2a99908e0542f71624b27bb20bb5b1"
Oct 30 13:41:21.731549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.731514    4993 scope.go:122] "RemoveContainer" containerID="f29dab413edcd80b978f262323de7528bd87b0e7b7f04baddff52fe73ab64396"
Oct 30 13:41:21.744164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.744137    4993 scope.go:122] "RemoveContainer" containerID="8c6652d7d26a2a59b5676e5604ea845a22f495a769c760ec3b05b0ab46a9e075"
Oct 30 13:41:21.873143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:21.873022    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"]
Oct 30 13:41:21.881304 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:41:21.881261    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22356426_47bd_40fb_b64a_df78bf31f32d.slice/crio-d6c0b19642c41a41f6090a98807507eb6554ae6d90fc286dbb700edac33a83e9 WatchSource:0}: Error finding container d6c0b19642c41a41f6090a98807507eb6554ae6d90fc286dbb700edac33a83e9: Status 404 returned error can't find the container with id d6c0b19642c41a41f6090a98807507eb6554ae6d90fc286dbb700edac33a83e9
Oct 30 13:41:22.210837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:22.210790    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z" event={"ID":"22356426-47bd-40fb-b64a-df78bf31f32d","Type":"ContainerStarted","Data":"d6c0b19642c41a41f6090a98807507eb6554ae6d90fc286dbb700edac33a83e9"}
Oct 30 13:41:23.222721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:23.222625    4993 generic.go:356] "Generic (PLEG): container finished" podID="22356426-47bd-40fb-b64a-df78bf31f32d" containerID="76171f413860bd56d93d7416841126bfea385974dc6cd47eed0540620f12b0eb" exitCode=0
Oct 30 13:41:23.222721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:23.222703    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z" event={"ID":"22356426-47bd-40fb-b64a-df78bf31f32d","Type":"ContainerDied","Data":"76171f413860bd56d93d7416841126bfea385974dc6cd47eed0540620f12b0eb"}
Oct 30 13:41:24.434324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.434264    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:24.556879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.556755    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-xks-ca\") pod \"22356426-47bd-40fb-b64a-df78bf31f32d\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") "
Oct 30 13:41:24.557086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.557062    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22356426-47bd-40fb-b64a-df78bf31f32d-kube-api-access-hx9c7\" (UniqueName: \"kubernetes.io/projected/22356426-47bd-40fb-b64a-df78bf31f32d-kube-api-access-hx9c7\") pod \"22356426-47bd-40fb-b64a-df78bf31f32d\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") "
Oct 30 13:41:24.557158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.557121    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-openshift-ca\") pod \"22356426-47bd-40fb-b64a-df78bf31f32d\" (UID: \"22356426-47bd-40fb-b64a-df78bf31f32d\") "
Oct 30 13:41:24.557550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.557522    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-openshift-ca" pod "22356426-47bd-40fb-b64a-df78bf31f32d" (UID: "22356426-47bd-40fb-b64a-df78bf31f32d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:24.558308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.558281    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:24.559620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.559596    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-xks-ca" pod "22356426-47bd-40fb-b64a-df78bf31f32d" (UID: "22356426-47bd-40fb-b64a-df78bf31f32d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:24.560292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.560265    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22356426-47bd-40fb-b64a-df78bf31f32d-kube-api-access-hx9c7" pod "22356426-47bd-40fb-b64a-df78bf31f32d" (UID: "22356426-47bd-40fb-b64a-df78bf31f32d"). InnerVolumeSpecName "kube-api-access-hx9c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:24.660579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.660537    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22356426-47bd-40fb-b64a-df78bf31f32d-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:24.660579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:24.660582    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9c7\" (UniqueName: \"kubernetes.io/projected/22356426-47bd-40fb-b64a-df78bf31f32d-kube-api-access-hx9c7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:25.248151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:25.248121    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"
Oct 30 13:41:25.248151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:25.248144    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z" event={"ID":"22356426-47bd-40fb-b64a-df78bf31f32d","Type":"ContainerDied","Data":"d6c0b19642c41a41f6090a98807507eb6554ae6d90fc286dbb700edac33a83e9"}
Oct 30 13:41:25.248436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:25.248262    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6c0b19642c41a41f6090a98807507eb6554ae6d90fc286dbb700edac33a83e9"
Oct 30 13:41:35.930457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:35.930413    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk" podUID="90039dac-eba4-4da5-854e-ed5e146756ef" containerName="main" containerID="cri-o://2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b" gracePeriod=45
Oct 30 13:41:36.265013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.264977    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:41:36.371313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.370748    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90039dac-eba4-4da5-854e-ed5e146756ef-kube-api-access-wq8xm\" (UniqueName: \"kubernetes.io/projected/90039dac-eba4-4da5-854e-ed5e146756ef-kube-api-access-wq8xm\") pod \"90039dac-eba4-4da5-854e-ed5e146756ef\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") "
Oct 30 13:41:36.371546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.371489    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-dshm\") pod \"90039dac-eba4-4da5-854e-ed5e146756ef\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") "
Oct 30 13:41:36.371618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.371553    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-home\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-home\") pod \"90039dac-eba4-4da5-854e-ed5e146756ef\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") "
Oct 30 13:41:36.371677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.371627    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90039dac-eba4-4da5-854e-ed5e146756ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/90039dac-eba4-4da5-854e-ed5e146756ef-tls-certs\") pod \"90039dac-eba4-4da5-854e-ed5e146756ef\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") "
Oct 30 13:41:36.371782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.371762    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-model-cache\") pod \"90039dac-eba4-4da5-854e-ed5e146756ef\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") "
Oct 30 13:41:36.371850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.371827    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-tmp-dir\") pod \"90039dac-eba4-4da5-854e-ed5e146756ef\" (UID: \"90039dac-eba4-4da5-854e-ed5e146756ef\") "
Oct 30 13:41:36.372929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.372893    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-model-cache" pod "90039dac-eba4-4da5-854e-ed5e146756ef" (UID: "90039dac-eba4-4da5-854e-ed5e146756ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:36.373077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.372973    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-tmp-dir" pod "90039dac-eba4-4da5-854e-ed5e146756ef" (UID: "90039dac-eba4-4da5-854e-ed5e146756ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:36.373077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.373033    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-home" pod "90039dac-eba4-4da5-854e-ed5e146756ef" (UID: "90039dac-eba4-4da5-854e-ed5e146756ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:36.374045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.374014    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90039dac-eba4-4da5-854e-ed5e146756ef-kube-api-access-wq8xm" pod "90039dac-eba4-4da5-854e-ed5e146756ef" (UID: "90039dac-eba4-4da5-854e-ed5e146756ef"). InnerVolumeSpecName "kube-api-access-wq8xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:36.375531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.375498    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90039dac-eba4-4da5-854e-ed5e146756ef-tls-certs" pod "90039dac-eba4-4da5-854e-ed5e146756ef" (UID: "90039dac-eba4-4da5-854e-ed5e146756ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:36.375797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.375776    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-dshm" pod "90039dac-eba4-4da5-854e-ed5e146756ef" (UID: "90039dac-eba4-4da5-854e-ed5e146756ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:36.437147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.437106    4993 generic.go:356] "Generic (PLEG): container finished" podID="90039dac-eba4-4da5-854e-ed5e146756ef" containerID="2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b" exitCode=0
Oct 30 13:41:36.437350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.437156    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"
Oct 30 13:41:36.437350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.437188    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk" event={"ID":"90039dac-eba4-4da5-854e-ed5e146756ef","Type":"ContainerDied","Data":"2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b"}
Oct 30 13:41:36.437350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.437250    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk" event={"ID":"90039dac-eba4-4da5-854e-ed5e146756ef","Type":"ContainerDied","Data":"9619c7bccaaa2e39c25812c9fccfb53df5dce9834bcc74aaae66586e0d1b6a64"}
Oct 30 13:41:36.437350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.437269    4993 scope.go:122] "RemoveContainer" containerID="2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b"
Oct 30 13:41:36.476837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.470400    4993 scope.go:122] "RemoveContainer" containerID="2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b"
Oct 30 13:41:36.501782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.491617    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90039dac-eba4-4da5-854e-ed5e146756ef-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:36.501782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.491667    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:36.501782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.491680    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:36.501782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.491694    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq8xm\" (UniqueName: \"kubernetes.io/projected/90039dac-eba4-4da5-854e-ed5e146756ef-kube-api-access-wq8xm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:36.501782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.491707    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:36.501782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.491719    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90039dac-eba4-4da5-854e-ed5e146756ef-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:41:36.501782 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:41:36.494381    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b\": container with ID starting with 2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b not found: ID does not exist" containerID="2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b"
Oct 30 13:41:36.501782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.494438    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b"} err="failed to get container status \"2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b\": rpc error: code = NotFound desc = could not find container \"2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b\": container with ID starting with 2509146839ce6215ff1ff237ec7a580194e645febc74384c9cf8004b89b52d6b not found: ID does not exist"
Oct 30 13:41:36.543267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.532303    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"]
Oct 30 13:41:36.543267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:36.540198    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ba646b/test-model-ba646b-kserve-56d86c4fd5-d7zsk"]
Oct 30 13:41:37.712237 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:41:37.710038    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90039dac-eba4-4da5-854e-ed5e146756ef" path="/var/lib/kubelet/pods/90039dac-eba4-4da5-854e-ed5e146756ef/volumes"
Oct 30 13:41:38.571510 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:41:38.571464    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:42:05.177227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:05.172285    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"]
Oct 30 13:42:05.222230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:05.204479    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d41320-7757z"]
Oct 30 13:42:05.694968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:05.694932    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22356426-47bd-40fb-b64a-df78bf31f32d" path="/var/lib/kubelet/pods/22356426-47bd-40fb-b64a-df78bf31f32d/volumes"
Oct 30 13:42:08.572056 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:42:08.572009    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:42:12.833866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:12.833826    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"]
Oct 30 13:42:12.834354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:12.834076    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy" containerID="cri-o://1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68" gracePeriod=30
Oct 30 13:42:13.294823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.294778    4993 patch_prober.go:29] interesting pod/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:42:13.295015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.294854    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:42:13.517084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.517036    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"]
Oct 30 13:42:13.517750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.517735    4993 cpu_manager.go:414] "containerMap: removing container" podUID="22356426-47bd-40fb-b64a-df78bf31f32d" containerName="revoke-keys"
Oct 30 13:42:13.517813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.517752    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22356426-47bd-40fb-b64a-df78bf31f32d" containerName="revoke-keys"
Oct 30 13:42:13.517813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.517786    4993 cpu_manager.go:414] "containerMap: removing container" podUID="90039dac-eba4-4da5-854e-ed5e146756ef" containerName="main"
Oct 30 13:42:13.517813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.517792    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90039dac-eba4-4da5-854e-ed5e146756ef" containerName="main"
Oct 30 13:42:13.517949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.517890    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="90039dac-eba4-4da5-854e-ed5e146756ef" containerName="main"
Oct 30 13:42:13.517949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.517906    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="22356426-47bd-40fb-b64a-df78bf31f32d" containerName="revoke-keys"
Oct 30 13:42:13.522173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.522143    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.530101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.530069    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"]
Oct 30 13:42:13.536979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.536941    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:13.537181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.537158    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:42:13.537181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.537172    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:42:13.537368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.537253    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:42:13.651510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.651404    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.651684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.651668    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.651726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.651709    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lskxt\" (UniqueName: \"kubernetes.io/projected/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-kube-api-access-lskxt\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.753714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.753670    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.753952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.753755    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lskxt\" (UniqueName: \"kubernetes.io/projected/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-kube-api-access-lskxt\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.754028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.753957    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.754691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.754666    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.756393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.756367    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.781181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.781143    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lskxt\" (UniqueName: \"kubernetes.io/projected/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-kube-api-access-lskxt\") pod \"maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:13.841195 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:13.841157    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:14.178447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:14.178415    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"]
Oct 30 13:42:14.876633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:14.876515    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp" event={"ID":"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f","Type":"ContainerStarted","Data":"3dd569881fbe5d65ee1c4eaf11e97891ec365030cddbab90b2897d701e9521bd"}
Oct 30 13:42:14.876633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:14.876580    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp" event={"ID":"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f","Type":"ContainerStarted","Data":"2175a761a1c7966edcdd1caecd2e8ba9a357d80c2cc33cb8b77c91d22e441a98"}
Oct 30 13:42:14.921002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:14.920939    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp" podStartSLOduration=1.92091893 podStartE2EDuration="1.92091893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:14.916728494 +0000 UTC m=+4199.392448104" watchObservedRunningTime="2026-10-30 13:42:14.92091893 +0000 UTC m=+4199.396638540"
Oct 30 13:42:15.916781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:15.916746    4993 generic.go:356] "Generic (PLEG): container finished" podID="3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f" containerID="3dd569881fbe5d65ee1c4eaf11e97891ec365030cddbab90b2897d701e9521bd" exitCode=0
Oct 30 13:42:15.917309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:15.916831    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp" event={"ID":"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f","Type":"ContainerDied","Data":"3dd569881fbe5d65ee1c4eaf11e97891ec365030cddbab90b2897d701e9521bd"}
Oct 30 13:42:16.478224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:16.478149    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:42:16.519695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:16.519665    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:42:17.101189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.101151    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:17.141332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.141300    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-kube-api-access-lskxt\" (UniqueName: \"kubernetes.io/projected/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-kube-api-access-lskxt\") pod \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") "
Oct 30 13:42:17.141524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.141439    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-xks-ca\") pod \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") "
Oct 30 13:42:17.141524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.141500    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-openshift-ca\") pod \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\" (UID: \"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f\") "
Oct 30 13:42:17.141886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.141855    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-openshift-ca" pod "3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f" (UID: "3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:17.142177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.142161    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:17.143556 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.143528    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-kube-api-access-lskxt" pod "3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f" (UID: "3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f"). InnerVolumeSpecName "kube-api-access-lskxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:17.143647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.143572    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-xks-ca" pod "3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f" (UID: "3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:17.243429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.243391    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lskxt\" (UniqueName: \"kubernetes.io/projected/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-kube-api-access-lskxt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:17.243429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.243425    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:17.935252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.935200    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp" event={"ID":"3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f","Type":"ContainerDied","Data":"2175a761a1c7966edcdd1caecd2e8ba9a357d80c2cc33cb8b77c91d22e441a98"}
Oct 30 13:42:17.935429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.935268    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2175a761a1c7966edcdd1caecd2e8ba9a357d80c2cc33cb8b77c91d22e441a98"
Oct 30 13:42:17.935429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:17.935268    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"
Oct 30 13:42:18.144940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.144910    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:42:18.258200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258163    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-podinfo\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.258437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258311    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-socket\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.258437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258336    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-data\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.258437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258361    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-token\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-token\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.258437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258384    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-credential-socket\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.258946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258498    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-envoy\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.258946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258570    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-kube-api-access-gxhfw\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-kube-api-access-gxhfw\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.258946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258740    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-certs\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.259410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.258991    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-data" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:18.259410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.259122    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istiod-ca-cert\") pod \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\" (UID: \"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a\") "
Oct 30 13:42:18.259570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.259399    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-socket" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:18.259649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.259623    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-credential-socket" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:18.259983 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.259956    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-certs" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:18.260384 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.260357    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istiod-ca-cert" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:18.260746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.260723    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.260869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.260854    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.260969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.260953    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.261070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.261056    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.261167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.261154    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.264076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.263752    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-token" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:18.264076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.264007    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-podinfo" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:42:18.265403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.264476    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-envoy" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:18.267747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.267724    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-kube-api-access-gxhfw" pod "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" (UID: "6c80a7fc-a3fa-46b8-8134-2bca5b383a6a"). InnerVolumeSpecName "kube-api-access-gxhfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:18.362953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.362899    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.362953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.362943    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.362953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.362957    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxhfw\" (UniqueName: \"kubernetes.io/projected/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-kube-api-access-gxhfw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.363225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.362971    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:42:18.945485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.945446    4993 generic.go:356] "Generic (PLEG): container finished" podID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerID="1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68" exitCode=0
Oct 30 13:42:18.945705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.945500    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"
Oct 30 13:42:18.945705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.945524    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" event={"ID":"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a","Type":"ContainerDied","Data":"1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68"}
Oct 30 13:42:18.945705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.945591    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq" event={"ID":"6c80a7fc-a3fa-46b8-8134-2bca5b383a6a","Type":"ContainerDied","Data":"228a16f075df8d101f46379c779e52af8e5e7297b82aa33a842c42f3bb0d47e6"}
Oct 30 13:42:18.945705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.945623    4993 scope.go:122] "RemoveContainer" containerID="1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68"
Oct 30 13:42:18.959576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.959273    4993 scope.go:122] "RemoveContainer" containerID="6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b"
Oct 30 13:42:18.975307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.975279    4993 scope.go:122] "RemoveContainer" containerID="1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68"
Oct 30 13:42:18.975675 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:42:18.975649    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68\": container with ID starting with 1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68 not found: ID does not exist" containerID="1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68"
Oct 30 13:42:18.975751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.975691    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68"} err="failed to get container status \"1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68\": rpc error: code = NotFound desc = could not find container \"1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68\": container with ID starting with 1cc6c01a6b5338a81fc572471d7628e640c31512b624d8e8e16814bdf57c8d68 not found: ID does not exist"
Oct 30 13:42:18.975751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.975718    4993 scope.go:122] "RemoveContainer" containerID="6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b"
Oct 30 13:42:18.975998 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:42:18.975978    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b\": container with ID starting with 6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b not found: ID does not exist" containerID="6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b"
Oct 30 13:42:18.976059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:18.976010    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b"} err="failed to get container status \"6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b\": rpc error: code = NotFound desc = could not find container \"6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b\": container with ID starting with 6114dd2f7c7e0245f7fe4e91cabc622730f662103928facb37aea92175acae0b not found: ID does not exist"
Oct 30 13:42:19.012288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:19.012248    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"]
Oct 30 13:42:19.023831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:19.023788    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d41320-openshift-default-9c8899ff9-hk9sq"]
Oct 30 13:42:19.694811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:42:19.694774    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" path="/var/lib/kubelet/pods/6c80a7fc-a3fa-46b8-8134-2bca5b383a6a/volumes"
Oct 30 13:42:38.575509 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:42:38.575437    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:43:08.573475 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:43:08.573434    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:43:14.836826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:14.836788    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"]
Oct 30 13:43:14.837332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:14.837000    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy" containerID="cri-o://304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0" gracePeriod=30
Oct 30 13:43:16.037235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:16.036622    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"]
Oct 30 13:43:16.059898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:16.059847    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ba646b-d49hp"]
Oct 30 13:43:16.162093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:16.162042    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"]
Oct 30 13:43:16.171041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:16.170998    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy" containerID="cri-o://150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039" gracePeriod=30
Oct 30 13:43:17.695451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:17.695407    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f" path="/var/lib/kubelet/pods/3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f/volumes"
Oct 30 13:43:20.148830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.148800    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 13:43:20.230561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230527    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-certs\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.230561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230564    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-envoy\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.230836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230651    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-data\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.230836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230820    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-credential-socket\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.230968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230864    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-token\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-token\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.230968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230893    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd38c30-f926-4a99-93fb-47b73375ef5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd38c30-f926-4a99-93fb-47b73375ef5e-istiod-ca-cert\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.230968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230923    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-certs" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.230968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230934    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-kube-api-access-zqjzs\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-kube-api-access-zqjzs\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.231220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.230983    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-socket\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.231220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.231065    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-podinfo\") pod \"dbd38c30-f926-4a99-93fb-47b73375ef5e\" (UID: \"dbd38c30-f926-4a99-93fb-47b73375ef5e\") "
Oct 30 13:43:20.231470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.231440    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd38c30-f926-4a99-93fb-47b73375ef5e-istiod-ca-cert" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:20.231567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.231509    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-data" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.231924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.231897    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.232095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.232079    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.232360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.232334    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd38c30-f926-4a99-93fb-47b73375ef5e-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.232485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.232130    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-socket" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.232485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.232338    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-credential-socket" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.234005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.233976    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-envoy" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.234150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.234132    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-token" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:20.234760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.234730    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-kube-api-access-zqjzs" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "kube-api-access-zqjzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:20.236243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.236198    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-podinfo" pod "dbd38c30-f926-4a99-93fb-47b73375ef5e" (UID: "dbd38c30-f926-4a99-93fb-47b73375ef5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:20.334039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.334000    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.334039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.334039    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.334369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.334051    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqjzs\" (UniqueName: \"kubernetes.io/projected/dbd38c30-f926-4a99-93fb-47b73375ef5e-kube-api-access-zqjzs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.334369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.334066    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.334369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.334078    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.334369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.334090    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd38c30-f926-4a99-93fb-47b73375ef5e-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.696401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.696360    4993 generic.go:356] "Generic (PLEG): container finished" podID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerID="304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0" exitCode=0
Oct 30 13:43:20.696626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.696431    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"
Oct 30 13:43:20.696626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.696479    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" event={"ID":"dbd38c30-f926-4a99-93fb-47b73375ef5e","Type":"ContainerDied","Data":"304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0"}
Oct 30 13:43:20.696626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.696539    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp" event={"ID":"dbd38c30-f926-4a99-93fb-47b73375ef5e","Type":"ContainerDied","Data":"b568aa7495f017f4e59e4e57977d06bd9a29735f80e838b9a2cf5e615f27688c"}
Oct 30 13:43:20.696626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.696569    4993 scope.go:122] "RemoveContainer" containerID="304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0"
Oct 30 13:43:20.710266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.710241    4993 scope.go:122] "RemoveContainer" containerID="81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2"
Oct 30 13:43:20.721834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.721813    4993 scope.go:122] "RemoveContainer" containerID="304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0"
Oct 30 13:43:20.722132 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:43:20.722113    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0\": container with ID starting with 304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0 not found: ID does not exist" containerID="304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0"
Oct 30 13:43:20.722185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.722144    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0"} err="failed to get container status \"304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0\": rpc error: code = NotFound desc = could not find container \"304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0\": container with ID starting with 304b8f38486bc5876d2f4135b0d4dc0e4ffd9cd301359aa8af88f42b88b4f9e0 not found: ID does not exist"
Oct 30 13:43:20.722185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.722162    4993 scope.go:122] "RemoveContainer" containerID="81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2"
Oct 30 13:43:20.722449 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:43:20.722429    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2\": container with ID starting with 81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2 not found: ID does not exist" containerID="81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2"
Oct 30 13:43:20.722509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.722457    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2"} err="failed to get container status \"81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2\": rpc error: code = NotFound desc = could not find container \"81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2\": container with ID starting with 81fd7e8d19911a5d64e2a4ed016fb4d13557824c1c46ac760909cfb9d05929d2 not found: ID does not exist"
Oct 30 13:43:20.750687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.750652    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"]
Oct 30 13:43:20.763199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:20.763166    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-9tjsp"]
Oct 30 13:43:21.642024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.641991    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 13:43:21.695927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.695874    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" path="/var/lib/kubelet/pods/dbd38c30-f926-4a99-93fb-47b73375ef5e/volumes"
Oct 30 13:43:21.710562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.710522    4993 generic.go:356] "Generic (PLEG): container finished" podID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerID="150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039" exitCode=0
Oct 30 13:43:21.710799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.710579    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"
Oct 30 13:43:21.710799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.710616    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" event={"ID":"6fb04db8-4e1d-4234-9519-647b5ac606d0","Type":"ContainerDied","Data":"150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039"}
Oct 30 13:43:21.710799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.710687    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx" event={"ID":"6fb04db8-4e1d-4234-9519-647b5ac606d0","Type":"ContainerDied","Data":"8ba0703180044f21824dbd8e533a6fa5b2ff9a3116f4a802e3e9ec1cda1aab37"}
Oct 30 13:43:21.710799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.710750    4993 scope.go:122] "RemoveContainer" containerID="150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039"
Oct 30 13:43:21.722044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722005    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-envoy\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.722249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722056    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-data\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.722249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722089    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-credential-socket\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.722249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722177    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fb04db8-4e1d-4234-9519-647b5ac606d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb04db8-4e1d-4234-9519-647b5ac606d0-istiod-ca-cert\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.722435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722360    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-socket\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.722435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722412    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-certs\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.722509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722480    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-kube-api-access-jdhbh\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-kube-api-access-jdhbh\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.722554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722522    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-podinfo\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.722597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.722574    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-token\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-token\") pod \"6fb04db8-4e1d-4234-9519-647b5ac606d0\" (UID: \"6fb04db8-4e1d-4234-9519-647b5ac606d0\") "
Oct 30 13:43:21.724125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.724074    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-socket" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:21.724823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.724704    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-certs" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:21.724823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.724773    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb04db8-4e1d-4234-9519-647b5ac606d0-istiod-ca-cert" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:21.724823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.724797    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-credential-socket" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:21.725539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.725504    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-data" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:21.726030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.725887    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-envoy" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:21.726151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.726108    4993 scope.go:122] "RemoveContainer" containerID="150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039"
Oct 30 13:43:21.728515 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:43:21.727409    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039\": container with ID starting with 150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039 not found: ID does not exist" containerID="150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039"
Oct 30 13:43:21.728515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.727462    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039"} err="failed to get container status \"150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039\": rpc error: code = NotFound desc = could not find container \"150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039\": container with ID starting with 150f65304d1d518c544b443031880c7bc5933db0845f0607dfe6291197794039 not found: ID does not exist"
Oct 30 13:43:21.728877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.728851    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-podinfo" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:21.729090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.729065    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-kube-api-access-jdhbh" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "kube-api-access-jdhbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:21.741428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.741392    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-token" pod "6fb04db8-4e1d-4234-9519-647b5ac606d0" (UID: "6fb04db8-4e1d-4234-9519-647b5ac606d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825660    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825694    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdhbh\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-kube-api-access-jdhbh\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825706    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825715    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825725    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825739    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825750    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825759    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb04db8-4e1d-4234-9519-647b5ac606d0-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.826638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:21.825770    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb04db8-4e1d-4234-9519-647b5ac606d0-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:22.064728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:22.064670    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"]
Oct 30 13:43:22.079722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:22.079673    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-gs4kx"]
Oct 30 13:43:23.700188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:23.700141    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" path="/var/lib/kubelet/pods/6fb04db8-4e1d-4234-9519-647b5ac606d0/volumes"
Oct 30 13:43:23.778234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:23.769951    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"]
Oct 30 13:43:23.778234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:23.770248    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy" containerID="cri-o://7017126add71196eef540c2924b20960e5247d57e1ba6cfc6a336cb438ef19b8" gracePeriod=30
Oct 30 13:43:28.946173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:28.946125    4993 generic.go:356] "Generic (PLEG): container finished" podID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerID="7017126add71196eef540c2924b20960e5247d57e1ba6cfc6a336cb438ef19b8" exitCode=0
Oct 30 13:43:28.946821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:28.946279    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" event={"ID":"4a44edeb-97be-4ef5-878c-7de3abd1bedd","Type":"ContainerDied","Data":"7017126add71196eef540c2924b20960e5247d57e1ba6cfc6a336cb438ef19b8"}
Oct 30 13:43:28.946821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:28.946373    4993 scope.go:122] "RemoveContainer" containerID="a8ceb1b41a76b585a16af3c45d43fc90dda8f5fdfbcad8287f25f21021694c0c"
Oct 30 13:43:29.114000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.113969    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:43:29.200444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200403    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-certs\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.200660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200462    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-token\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-token\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.200660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200501    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istiod-ca-cert\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.200660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200589    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-kube-api-access-4d9jj\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-kube-api-access-4d9jj\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.200855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200698    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-certs" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:29.200855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200815    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-credential-socket\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.200967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200884    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-data\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.200967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200956    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istiod-ca-cert" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:29.201098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.200980    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-podinfo\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.201098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201065    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-socket\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.201241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201125    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-envoy\") pod \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\" (UID: \"4a44edeb-97be-4ef5-878c-7de3abd1bedd\") "
Oct 30 13:43:29.201241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201219    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-data" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:29.201413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201371    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-socket" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:29.201589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201563    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-credential-socket" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:29.201987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201892    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.201987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201923    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.201987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201934    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.201987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201943    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.201987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.201954    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.203892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.203861    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-envoy" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:29.207042 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.206977    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-podinfo" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:29.207627 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.207585    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-token" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:29.227505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.227458    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-kube-api-access-4d9jj" pod "4a44edeb-97be-4ef5-878c-7de3abd1bedd" (UID: "4a44edeb-97be-4ef5-878c-7de3abd1bedd"). InnerVolumeSpecName "kube-api-access-4d9jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:29.303599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.303556    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d9jj\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-kube-api-access-4d9jj\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.303599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.303596    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.303599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.303606    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.303881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.303614    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a44edeb-97be-4ef5-878c-7de3abd1bedd-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.960791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.960691    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"
Oct 30 13:43:29.960791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.960732    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz" event={"ID":"4a44edeb-97be-4ef5-878c-7de3abd1bedd","Type":"ContainerDied","Data":"9e628b314196ec40a04a5c70d6a86031c746999ed674f57aca216f85ccf42150"}
Oct 30 13:43:29.961412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:29.960805    4993 scope.go:122] "RemoveContainer" containerID="7017126add71196eef540c2924b20960e5247d57e1ba6cfc6a336cb438ef19b8"
Oct 30 13:43:30.026902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:30.026855    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"]
Oct 30 13:43:30.037887 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:30.037845    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-qqbsz"]
Oct 30 13:43:31.693700 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:31.693654    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" path="/var/lib/kubelet/pods/4a44edeb-97be-4ef5-878c-7de3abd1bedd/volumes"
Oct 30 13:43:38.572687 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:43:38.572634    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:43:50.682921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.682876    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"]
Oct 30 13:43:50.683844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683810    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy"
Oct 30 13:43:50.683844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683838    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy"
Oct 30 13:43:50.683940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683859    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy"
Oct 30 13:43:50.683940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683867    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy"
Oct 30 13:43:50.683940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683888    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f" containerName="revoke-keys"
Oct 30 13:43:50.683940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683898    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f" containerName="revoke-keys"
Oct 30 13:43:50.683940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683921    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy"
Oct 30 13:43:50.683940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683928    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy"
Oct 30 13:43:50.684141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683963    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy"
Oct 30 13:43:50.684141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.683970    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy"
Oct 30 13:43:50.684141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684010    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy"
Oct 30 13:43:50.684141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684018    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy"
Oct 30 13:43:50.684307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684156    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy"
Oct 30 13:43:50.684307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684169    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb04db8-4e1d-4234-9519-647b5ac606d0" containerName="istio-proxy"
Oct 30 13:43:50.684307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684181    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3deffcd5-2fbd-4be8-abb5-4b4bb07dd64f" containerName="revoke-keys"
Oct 30 13:43:50.684307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684193    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy"
Oct 30 13:43:50.684307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684219    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy"
Oct 30 13:43:50.684307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684229    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy"
Oct 30 13:43:50.684307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.684243    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd38c30-f926-4a99-93fb-47b73375ef5e" containerName="istio-proxy"
Oct 30 13:43:50.690094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.690060    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.716622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.716581    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:43:50.716622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.716581    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:43:50.716622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.716581    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:43:50.717089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.716809    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:43:50.723659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.723617    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"]
Oct 30 13:43:50.771674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.771634    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th4pt\" (UniqueName: \"kubernetes.io/projected/19405398-30b1-45b7-a28e-54c223cf12c5-kube-api-access-th4pt\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.771888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.771757    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.771968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.771949    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.873735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.873679    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4pt\" (UniqueName: \"kubernetes.io/projected/19405398-30b1-45b7-a28e-54c223cf12c5-kube-api-access-th4pt\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.873967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.873780    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.873967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.873928    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.874662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.874636    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.876495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.876465    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:50.898074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:50.898039    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th4pt\" (UniqueName: \"kubernetes.io/projected/19405398-30b1-45b7-a28e-54c223cf12c5-kube-api-access-th4pt\") pod \"maas-api-revoke-sub-12f2c996-jpjzm\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:51.015318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:51.010519    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:51.348286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:51.348179    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"]
Oct 30 13:43:52.162886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:52.162763    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm" event={"ID":"19405398-30b1-45b7-a28e-54c223cf12c5","Type":"ContainerStarted","Data":"3805f25351a8ed2d0b80a8db62772e8af8b0abbf7ac3bd3df2822af1840aea5d"}
Oct 30 13:43:52.163396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:52.162883    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm" event={"ID":"19405398-30b1-45b7-a28e-54c223cf12c5","Type":"ContainerStarted","Data":"f07f9c53298d5ae7982e7acf4955747524e93696a63d35d95126bd03690875bf"}
Oct 30 13:43:52.206767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:52.206694    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm" podStartSLOduration=2.206677744 podStartE2EDuration="2.206677744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:52.205567311 +0000 UTC m=+4296.681286922" watchObservedRunningTime="2026-10-30 13:43:52.206677744 +0000 UTC m=+4296.682397353"
Oct 30 13:43:53.173600 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:53.173558    4993 generic.go:356] "Generic (PLEG): container finished" podID="19405398-30b1-45b7-a28e-54c223cf12c5" containerID="3805f25351a8ed2d0b80a8db62772e8af8b0abbf7ac3bd3df2822af1840aea5d" exitCode=0
Oct 30 13:43:53.174048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:53.173644    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm" event={"ID":"19405398-30b1-45b7-a28e-54c223cf12c5","Type":"ContainerDied","Data":"3805f25351a8ed2d0b80a8db62772e8af8b0abbf7ac3bd3df2822af1840aea5d"}
Oct 30 13:43:54.507273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.503659    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"]
Oct 30 13:43:54.507273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.504589    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy"
Oct 30 13:43:54.507273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.504609    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy"
Oct 30 13:43:54.507273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.504646    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy"
Oct 30 13:43:54.507273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.504656    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a44edeb-97be-4ef5-878c-7de3abd1bedd" containerName="istio-proxy"
Oct 30 13:43:54.507273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.504814    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c80a7fc-a3fa-46b8-8134-2bca5b383a6a" containerName="istio-proxy"
Oct 30 13:43:54.509578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.509521    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.533227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.529896    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"]
Oct 30 13:43:54.577629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.577544    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:54.585225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.584459    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-xks-ca\") pod \"19405398-30b1-45b7-a28e-54c223cf12c5\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") "
Oct 30 13:43:54.585225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.584556    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19405398-30b1-45b7-a28e-54c223cf12c5-kube-api-access-th4pt\" (UniqueName: \"kubernetes.io/projected/19405398-30b1-45b7-a28e-54c223cf12c5-kube-api-access-th4pt\") pod \"19405398-30b1-45b7-a28e-54c223cf12c5\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") "
Oct 30 13:43:54.585225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.585100    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-openshift-ca\") pod \"19405398-30b1-45b7-a28e-54c223cf12c5\" (UID: \"19405398-30b1-45b7-a28e-54c223cf12c5\") "
Oct 30 13:43:54.585839 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.585671    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.585977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.585904    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.586291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.586066    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9x7\" (UniqueName: \"kubernetes.io/projected/833a70fb-536c-4120-b052-9602e8ebe0ac-kube-api-access-rs9x7\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.586561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.586524    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-openshift-ca" pod "19405398-30b1-45b7-a28e-54c223cf12c5" (UID: "19405398-30b1-45b7-a28e-54c223cf12c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:54.593909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.587805    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-xks-ca" pod "19405398-30b1-45b7-a28e-54c223cf12c5" (UID: "19405398-30b1-45b7-a28e-54c223cf12c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:54.594341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.594291    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19405398-30b1-45b7-a28e-54c223cf12c5-kube-api-access-th4pt" pod "19405398-30b1-45b7-a28e-54c223cf12c5" (UID: "19405398-30b1-45b7-a28e-54c223cf12c5"). InnerVolumeSpecName "kube-api-access-th4pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:54.688733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.688685    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.688908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.688812    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9x7\" (UniqueName: \"kubernetes.io/projected/833a70fb-536c-4120-b052-9602e8ebe0ac-kube-api-access-rs9x7\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.689008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.688990    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.689176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.689157    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:54.689313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.689181    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th4pt\" (UniqueName: \"kubernetes.io/projected/19405398-30b1-45b7-a28e-54c223cf12c5-kube-api-access-th4pt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:54.689313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.689196    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19405398-30b1-45b7-a28e-54c223cf12c5-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:54.690423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.690394    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.692272 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.692242    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.713391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.713351    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9x7\" (UniqueName: \"kubernetes.io/projected/833a70fb-536c-4120-b052-9602e8ebe0ac-kube-api-access-rs9x7\") pod \"maas-api-revoke-sub-b99dd107-j7lqt\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:54.881091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:54.880751    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:55.205290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:55.204545    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm" event={"ID":"19405398-30b1-45b7-a28e-54c223cf12c5","Type":"ContainerDied","Data":"f07f9c53298d5ae7982e7acf4955747524e93696a63d35d95126bd03690875bf"}
Oct 30 13:43:55.205290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:55.204617    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f07f9c53298d5ae7982e7acf4955747524e93696a63d35d95126bd03690875bf"
Oct 30 13:43:55.205290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:55.204694    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"
Oct 30 13:43:55.240845 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:43:55.240753    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833a70fb_536c_4120_b052_9602e8ebe0ac.slice/crio-184914d9898007e2062be1c0e3d93d8dbd73432081f128fe11a5eb8fc47b54fe WatchSource:0}: Error finding container 184914d9898007e2062be1c0e3d93d8dbd73432081f128fe11a5eb8fc47b54fe: Status 404 returned error can't find the container with id 184914d9898007e2062be1c0e3d93d8dbd73432081f128fe11a5eb8fc47b54fe
Oct 30 13:43:55.243150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:55.242992    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"]
Oct 30 13:43:56.266845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:56.266794    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt" event={"ID":"833a70fb-536c-4120-b052-9602e8ebe0ac","Type":"ContainerStarted","Data":"71e455212a767156494b0bbce28b2e9f5db0e3ac2a751e477b1a53529941ab67"}
Oct 30 13:43:56.267602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:56.267568    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt" event={"ID":"833a70fb-536c-4120-b052-9602e8ebe0ac","Type":"ContainerStarted","Data":"184914d9898007e2062be1c0e3d93d8dbd73432081f128fe11a5eb8fc47b54fe"}
Oct 30 13:43:56.315810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:56.315738    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt" podStartSLOduration=2.315717248 podStartE2EDuration="2.315717248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:56.314945279 +0000 UTC m=+4300.790664923" watchObservedRunningTime="2026-10-30 13:43:56.315717248 +0000 UTC m=+4300.791436861"
Oct 30 13:43:57.297231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:57.293174    4993 generic.go:356] "Generic (PLEG): container finished" podID="833a70fb-536c-4120-b052-9602e8ebe0ac" containerID="71e455212a767156494b0bbce28b2e9f5db0e3ac2a751e477b1a53529941ab67" exitCode=0
Oct 30 13:43:57.297231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:57.293374    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt" event={"ID":"833a70fb-536c-4120-b052-9602e8ebe0ac","Type":"ContainerDied","Data":"71e455212a767156494b0bbce28b2e9f5db0e3ac2a751e477b1a53529941ab67"}
Oct 30 13:43:58.704239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.700500    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:58.736189 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.736143    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-xks-ca\") pod \"833a70fb-536c-4120-b052-9602e8ebe0ac\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") "
Oct 30 13:43:58.736425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.736308    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-openshift-ca\") pod \"833a70fb-536c-4120-b052-9602e8ebe0ac\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") "
Oct 30 13:43:58.736425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.736381    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833a70fb-536c-4120-b052-9602e8ebe0ac-kube-api-access-rs9x7\" (UniqueName: \"kubernetes.io/projected/833a70fb-536c-4120-b052-9602e8ebe0ac-kube-api-access-rs9x7\") pod \"833a70fb-536c-4120-b052-9602e8ebe0ac\" (UID: \"833a70fb-536c-4120-b052-9602e8ebe0ac\") "
Oct 30 13:43:58.736748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.736715    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-openshift-ca" pod "833a70fb-536c-4120-b052-9602e8ebe0ac" (UID: "833a70fb-536c-4120-b052-9602e8ebe0ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:58.737501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.737193    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:58.738927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.738898    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-xks-ca" pod "833a70fb-536c-4120-b052-9602e8ebe0ac" (UID: "833a70fb-536c-4120-b052-9602e8ebe0ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:58.739073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.739047    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833a70fb-536c-4120-b052-9602e8ebe0ac-kube-api-access-rs9x7" pod "833a70fb-536c-4120-b052-9602e8ebe0ac" (UID: "833a70fb-536c-4120-b052-9602e8ebe0ac"). InnerVolumeSpecName "kube-api-access-rs9x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:58.839075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.839034    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9x7\" (UniqueName: \"kubernetes.io/projected/833a70fb-536c-4120-b052-9602e8ebe0ac-kube-api-access-rs9x7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:58.839075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:58.839073    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833a70fb-536c-4120-b052-9602e8ebe0ac-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:43:59.332819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:59.332781    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"
Oct 30 13:43:59.332819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:59.332791    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt" event={"ID":"833a70fb-536c-4120-b052-9602e8ebe0ac","Type":"ContainerDied","Data":"184914d9898007e2062be1c0e3d93d8dbd73432081f128fe11a5eb8fc47b54fe"}
Oct 30 13:43:59.333048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:43:59.332848    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="184914d9898007e2062be1c0e3d93d8dbd73432081f128fe11a5eb8fc47b54fe"
Oct 30 13:44:02.207164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.207118    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"]
Oct 30 13:44:02.207922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.207898    4993 cpu_manager.go:414] "containerMap: removing container" podUID="19405398-30b1-45b7-a28e-54c223cf12c5" containerName="revoke-keys"
Oct 30 13:44:02.207978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.207925    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19405398-30b1-45b7-a28e-54c223cf12c5" containerName="revoke-keys"
Oct 30 13:44:02.207978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.207956    4993 cpu_manager.go:414] "containerMap: removing container" podUID="833a70fb-536c-4120-b052-9602e8ebe0ac" containerName="revoke-keys"
Oct 30 13:44:02.207978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.207965    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833a70fb-536c-4120-b052-9602e8ebe0ac" containerName="revoke-keys"
Oct 30 13:44:02.208170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.208157    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="19405398-30b1-45b7-a28e-54c223cf12c5" containerName="revoke-keys"
Oct 30 13:44:02.208254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.208175    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="833a70fb-536c-4120-b052-9602e8ebe0ac" containerName="revoke-keys"
Oct 30 13:44:02.212623 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.212600    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.242571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.242532    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:44:02.242571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.242543    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:44:02.242571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.242551    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:44:02.242897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.242581    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:02.246720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.246682    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"]
Oct 30 13:44:02.305886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.305844    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcnx\" (UniqueName: \"kubernetes.io/projected/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-kube-api-access-qxcnx\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.306109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.305904    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.306109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.306019    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.407768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.407730    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.407997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.407790    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.408134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.408115    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcnx\" (UniqueName: \"kubernetes.io/projected/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-kube-api-access-qxcnx\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.408471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.408449    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.410511 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.410470    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.431673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.431635    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcnx\" (UniqueName: \"kubernetes.io/projected/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-kube-api-access-qxcnx\") pod \"maas-api-revoke-sub-6ece70c9-zbmx5\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.527999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.527967    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:02.849161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:02.849067    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"]
Oct 30 13:44:02.853817 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:02.853783    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44c27aa_73bc_490f_9865_eb4fb8f42bbd.slice/crio-fefac0f7258d7012901acf28f2785567df66fd9d982cf2a7c4fa1aed2bb9ecbe WatchSource:0}: Error finding container fefac0f7258d7012901acf28f2785567df66fd9d982cf2a7c4fa1aed2bb9ecbe: Status 404 returned error can't find the container with id fefac0f7258d7012901acf28f2785567df66fd9d982cf2a7c4fa1aed2bb9ecbe
Oct 30 13:44:03.370793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:03.370751    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5" event={"ID":"f44c27aa-73bc-490f-9865-eb4fb8f42bbd","Type":"ContainerStarted","Data":"4d94a75fedfa7f2d38bdf1a54c2b923598a14eaeaee7c9e3b6ccdbf55b4fa646"}
Oct 30 13:44:03.371299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:03.370816    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5" event={"ID":"f44c27aa-73bc-490f-9865-eb4fb8f42bbd","Type":"ContainerStarted","Data":"fefac0f7258d7012901acf28f2785567df66fd9d982cf2a7c4fa1aed2bb9ecbe"}
Oct 30 13:44:03.417066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:03.416938    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5" podStartSLOduration=1.4169176540000001 podStartE2EDuration="1.416917654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:03.41248693 +0000 UTC m=+4307.888206540" watchObservedRunningTime="2026-10-30 13:44:03.416917654 +0000 UTC m=+4307.892637265"
Oct 30 13:44:04.382105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:04.382065    4993 generic.go:356] "Generic (PLEG): container finished" podID="f44c27aa-73bc-490f-9865-eb4fb8f42bbd" containerID="4d94a75fedfa7f2d38bdf1a54c2b923598a14eaeaee7c9e3b6ccdbf55b4fa646" exitCode=0
Oct 30 13:44:04.382589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:04.382144    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5" event={"ID":"f44c27aa-73bc-490f-9865-eb4fb8f42bbd","Type":"ContainerDied","Data":"4d94a75fedfa7f2d38bdf1a54c2b923598a14eaeaee7c9e3b6ccdbf55b4fa646"}
Oct 30 13:44:05.566139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.566109    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:05.667974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.667885    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-xks-ca\") pod \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") "
Oct 30 13:44:05.667974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.667938    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-kube-api-access-qxcnx\" (UniqueName: \"kubernetes.io/projected/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-kube-api-access-qxcnx\") pod \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") "
Oct 30 13:44:05.668199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.668186    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-openshift-ca\") pod \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\" (UID: \"f44c27aa-73bc-490f-9865-eb4fb8f42bbd\") "
Oct 30 13:44:05.668515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.668490    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-openshift-ca" pod "f44c27aa-73bc-490f-9865-eb4fb8f42bbd" (UID: "f44c27aa-73bc-490f-9865-eb4fb8f42bbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:05.668932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.668904    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:05.670282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.670257    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-xks-ca" pod "f44c27aa-73bc-490f-9865-eb4fb8f42bbd" (UID: "f44c27aa-73bc-490f-9865-eb4fb8f42bbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:05.670371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.670349    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-kube-api-access-qxcnx" pod "f44c27aa-73bc-490f-9865-eb4fb8f42bbd" (UID: "f44c27aa-73bc-490f-9865-eb4fb8f42bbd"). InnerVolumeSpecName "kube-api-access-qxcnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:05.770536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.770498    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:05.770536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:05.770529    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxcnx\" (UniqueName: \"kubernetes.io/projected/f44c27aa-73bc-490f-9865-eb4fb8f42bbd-kube-api-access-qxcnx\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:06.403473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:06.403423    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5" event={"ID":"f44c27aa-73bc-490f-9865-eb4fb8f42bbd","Type":"ContainerDied","Data":"fefac0f7258d7012901acf28f2785567df66fd9d982cf2a7c4fa1aed2bb9ecbe"}
Oct 30 13:44:06.403473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:06.403461    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"
Oct 30 13:44:06.403731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:06.403499    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fefac0f7258d7012901acf28f2785567df66fd9d982cf2a7c4fa1aed2bb9ecbe"
Oct 30 13:44:08.575362 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:44:08.575309    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:44:22.211105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.211051    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"]
Oct 30 13:44:22.212171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.212061    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f44c27aa-73bc-490f-9865-eb4fb8f42bbd" containerName="revoke-keys"
Oct 30 13:44:22.212382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.212363    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44c27aa-73bc-490f-9865-eb4fb8f42bbd" containerName="revoke-keys"
Oct 30 13:44:22.212807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.212793    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44c27aa-73bc-490f-9865-eb4fb8f42bbd" containerName="revoke-keys"
Oct 30 13:44:22.219937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.219900    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.264676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.258845    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:44:22.264676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.258944    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:22.350429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.334944    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:44:22.350429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.335032    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:44:22.437257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.403780    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.437257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.403866    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.437257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.404067    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xl2\" (UniqueName: \"kubernetes.io/projected/6f5ec855-5930-44b7-9218-3449d29dd739-kube-api-access-86xl2\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.437257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.430179    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"]
Oct 30 13:44:22.467507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.467446    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"]
Oct 30 13:44:22.475854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.475773    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.507231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.506759    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"]
Oct 30 13:44:22.509543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.507860    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.509543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.507928    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.509543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.508136    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xl2\" (UniqueName: \"kubernetes.io/projected/6f5ec855-5930-44b7-9218-3449d29dd739-kube-api-access-86xl2\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.509543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.509151    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.518844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.518740    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.542235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.539333    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xl2\" (UniqueName: \"kubernetes.io/projected/6f5ec855-5930-44b7-9218-3449d29dd739-kube-api-access-86xl2\") pod \"maas-api-revoke-sub-6d1f3bc1-d8z67\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.609226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.609169    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52d7p\" (UniqueName: \"kubernetes.io/projected/ba2d2426-ac66-4791-9152-032cad4046c3-kube-api-access-52d7p\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.609426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.609308    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.609471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.609417    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.711437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.711380    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.711729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.711708    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52d7p\" (UniqueName: \"kubernetes.io/projected/ba2d2426-ac66-4791-9152-032cad4046c3-kube-api-access-52d7p\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.711822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.711754    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.712476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.712446    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.714025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.714000    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.730684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.730649    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:22.735467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.735427    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52d7p\" (UniqueName: \"kubernetes.io/projected/ba2d2426-ac66-4791-9152-032cad4046c3-kube-api-access-52d7p\") pod \"maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:22.825520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:22.825274    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:23.047726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:23.047640    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"]
Oct 30 13:44:23.047863 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:23.047784    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5ec855_5930_44b7_9218_3449d29dd739.slice/crio-8a6939a53c7570e92047a13bfd1a715832c3527e8f253a8843a50245cd230eb3 WatchSource:0}: Error finding container 8a6939a53c7570e92047a13bfd1a715832c3527e8f253a8843a50245cd230eb3: Status 404 returned error can't find the container with id 8a6939a53c7570e92047a13bfd1a715832c3527e8f253a8843a50245cd230eb3
Oct 30 13:44:23.178542 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:23.178493    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2d2426_ac66_4791_9152_032cad4046c3.slice/crio-525bf36945805c21f6253672033833a9ef608f592471301770b2a317367ea009 WatchSource:0}: Error finding container 525bf36945805c21f6253672033833a9ef608f592471301770b2a317367ea009: Status 404 returned error can't find the container with id 525bf36945805c21f6253672033833a9ef608f592471301770b2a317367ea009
Oct 30 13:44:23.178733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:23.178703    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"]
Oct 30 13:44:23.599338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:23.599280    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm" event={"ID":"ba2d2426-ac66-4791-9152-032cad4046c3","Type":"ContainerStarted","Data":"9bd390d8d4c61416f0b384faf3a9c9a58b157732b2632cfd8b600300336d5a6a"}
Oct 30 13:44:23.599928 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:23.599347    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm" event={"ID":"ba2d2426-ac66-4791-9152-032cad4046c3","Type":"ContainerStarted","Data":"525bf36945805c21f6253672033833a9ef608f592471301770b2a317367ea009"}
Oct 30 13:44:23.601432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:23.601391    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67" event={"ID":"6f5ec855-5930-44b7-9218-3449d29dd739","Type":"ContainerStarted","Data":"820d0b7fd974d0078c541cbef8314822535c51c23ad0a32890c5293235fbc1bb"}
Oct 30 13:44:23.601591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:23.601451    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67" event={"ID":"6f5ec855-5930-44b7-9218-3449d29dd739","Type":"ContainerStarted","Data":"8a6939a53c7570e92047a13bfd1a715832c3527e8f253a8843a50245cd230eb3"}
Oct 30 13:44:23.639639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:23.639530    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm" podStartSLOduration=1.639517642 podStartE2EDuration="1.639517642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:23.63900843 +0000 UTC m=+4328.114728039" watchObservedRunningTime="2026-10-30 13:44:23.639517642 +0000 UTC m=+4328.115237251"
Oct 30 13:44:23.682006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:23.681937    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67" podStartSLOduration=1.681919933 podStartE2EDuration="1.681919933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:23.679050091 +0000 UTC m=+4328.154769702" watchObservedRunningTime="2026-10-30 13:44:23.681919933 +0000 UTC m=+4328.157639545"
Oct 30 13:44:24.615699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:24.615663    4993 generic.go:356] "Generic (PLEG): container finished" podID="6f5ec855-5930-44b7-9218-3449d29dd739" containerID="820d0b7fd974d0078c541cbef8314822535c51c23ad0a32890c5293235fbc1bb" exitCode=0
Oct 30 13:44:24.616182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:24.615753    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67" event={"ID":"6f5ec855-5930-44b7-9218-3449d29dd739","Type":"ContainerDied","Data":"820d0b7fd974d0078c541cbef8314822535c51c23ad0a32890c5293235fbc1bb"}
Oct 30 13:44:24.618133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:24.618104    4993 generic.go:356] "Generic (PLEG): container finished" podID="ba2d2426-ac66-4791-9152-032cad4046c3" containerID="9bd390d8d4c61416f0b384faf3a9c9a58b157732b2632cfd8b600300336d5a6a" exitCode=0
Oct 30 13:44:24.618342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:24.618248    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm" event={"ID":"ba2d2426-ac66-4791-9152-032cad4046c3","Type":"ContainerDied","Data":"9bd390d8d4c61416f0b384faf3a9c9a58b157732b2632cfd8b600300336d5a6a"}
Oct 30 13:44:25.959275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:25.959180    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:26.100061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.096733    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-openshift-ca\") pod \"ba2d2426-ac66-4791-9152-032cad4046c3\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") "
Oct 30 13:44:26.100061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.096789    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2d2426-ac66-4791-9152-032cad4046c3-kube-api-access-52d7p\" (UniqueName: \"kubernetes.io/projected/ba2d2426-ac66-4791-9152-032cad4046c3-kube-api-access-52d7p\") pod \"ba2d2426-ac66-4791-9152-032cad4046c3\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") "
Oct 30 13:44:26.100061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.096958    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-xks-ca\") pod \"ba2d2426-ac66-4791-9152-032cad4046c3\" (UID: \"ba2d2426-ac66-4791-9152-032cad4046c3\") "
Oct 30 13:44:26.118269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.104362    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-openshift-ca" pod "ba2d2426-ac66-4791-9152-032cad4046c3" (UID: "ba2d2426-ac66-4791-9152-032cad4046c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:26.192336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.188159    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2d2426-ac66-4791-9152-032cad4046c3-kube-api-access-52d7p" pod "ba2d2426-ac66-4791-9152-032cad4046c3" (UID: "ba2d2426-ac66-4791-9152-032cad4046c3"). InnerVolumeSpecName "kube-api-access-52d7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:26.192779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.192738    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-xks-ca" pod "ba2d2426-ac66-4791-9152-032cad4046c3" (UID: "ba2d2426-ac66-4791-9152-032cad4046c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:26.212091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.210510    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:26.212091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.210553    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52d7p\" (UniqueName: \"kubernetes.io/projected/ba2d2426-ac66-4791-9152-032cad4046c3-kube-api-access-52d7p\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:26.212091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.210571    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba2d2426-ac66-4791-9152-032cad4046c3-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:26.225236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.221182    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:26.316934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.311743    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5ec855-5930-44b7-9218-3449d29dd739-kube-api-access-86xl2\" (UniqueName: \"kubernetes.io/projected/6f5ec855-5930-44b7-9218-3449d29dd739-kube-api-access-86xl2\") pod \"6f5ec855-5930-44b7-9218-3449d29dd739\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") "
Oct 30 13:44:26.316934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.311912    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-openshift-ca\") pod \"6f5ec855-5930-44b7-9218-3449d29dd739\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") "
Oct 30 13:44:26.316934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.312094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-xks-ca\") pod \"6f5ec855-5930-44b7-9218-3449d29dd739\" (UID: \"6f5ec855-5930-44b7-9218-3449d29dd739\") "
Oct 30 13:44:26.316934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.313741    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-openshift-ca" pod "6f5ec855-5930-44b7-9218-3449d29dd739" (UID: "6f5ec855-5930-44b7-9218-3449d29dd739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:26.384232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.377741    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-xks-ca" pod "6f5ec855-5930-44b7-9218-3449d29dd739" (UID: "6f5ec855-5930-44b7-9218-3449d29dd739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:26.447234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.420751    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:26.447234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.442638    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5ec855-5930-44b7-9218-3449d29dd739-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:26.447234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.420799    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5ec855-5930-44b7-9218-3449d29dd739-kube-api-access-86xl2" pod "6f5ec855-5930-44b7-9218-3449d29dd739" (UID: "6f5ec855-5930-44b7-9218-3449d29dd739"). InnerVolumeSpecName "kube-api-access-86xl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:26.547229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.546429    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86xl2\" (UniqueName: \"kubernetes.io/projected/6f5ec855-5930-44b7-9218-3449d29dd739-kube-api-access-86xl2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:26.814284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.801704    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67" event={"ID":"6f5ec855-5930-44b7-9218-3449d29dd739","Type":"ContainerDied","Data":"8a6939a53c7570e92047a13bfd1a715832c3527e8f253a8843a50245cd230eb3"}
Oct 30 13:44:26.814284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.801815    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a6939a53c7570e92047a13bfd1a715832c3527e8f253a8843a50245cd230eb3"
Oct 30 13:44:26.814284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.801892    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"
Oct 30 13:44:26.824177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.824124    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm" event={"ID":"ba2d2426-ac66-4791-9152-032cad4046c3","Type":"ContainerDied","Data":"525bf36945805c21f6253672033833a9ef608f592471301770b2a317367ea009"}
Oct 30 13:44:26.824807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.824460    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="525bf36945805c21f6253672033833a9ef608f592471301770b2a317367ea009"
Oct 30 13:44:26.824976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:26.824670    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"
Oct 30 13:44:30.808229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.807329    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"]
Oct 30 13:44:30.808812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.808334    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ba2d2426-ac66-4791-9152-032cad4046c3" containerName="revoke-keys"
Oct 30 13:44:30.808812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.808359    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2d2426-ac66-4791-9152-032cad4046c3" containerName="revoke-keys"
Oct 30 13:44:30.808812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.808401    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6f5ec855-5930-44b7-9218-3449d29dd739" containerName="revoke-keys"
Oct 30 13:44:30.808812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.808410    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5ec855-5930-44b7-9218-3449d29dd739" containerName="revoke-keys"
Oct 30 13:44:30.808812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.808604    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2d2426-ac66-4791-9152-032cad4046c3" containerName="revoke-keys"
Oct 30 13:44:30.808812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.808623    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5ec855-5930-44b7-9218-3449d29dd739" containerName="revoke-keys"
Oct 30 13:44:30.844267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.839073    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"]
Oct 30 13:44:30.844267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.839228    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.870715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.870653    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.871072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.871045    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.871392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.871364    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.871575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.871554    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.871778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.871758    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6zsm\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-kube-api-access-v6zsm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.871933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.871911    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.872067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.872051    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.872252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.872233    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.872432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.872413    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.973703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.973662    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.973903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.973732    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zsm\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-kube-api-access-v6zsm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.973903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.973761    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.973981    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974038    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974082    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974265    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974357    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974461    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974631    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974684    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974813    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.974968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.974844    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.975294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.975270    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.976711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.976687    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.976832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.976757    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.997397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.997358    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:30.997560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:30.997403    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6zsm\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-kube-api-access-v6zsm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-f8kbm\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:31.213816 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:31.213702    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:31.559145 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:31.559104    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"]
Oct 30 13:44:31.559910 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:31.559874    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a75412_1fd2_4c36_a3b3_4d87736d48a4.slice/crio-c1029597e3e9575287135e23733b7110fce1f66231894c60203e3f15b38c2413 WatchSource:0}: Error finding container c1029597e3e9575287135e23733b7110fce1f66231894c60203e3f15b38c2413: Status 404 returned error can't find the container with id c1029597e3e9575287135e23733b7110fce1f66231894c60203e3f15b38c2413
Oct 30 13:44:31.563246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:31.563192    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:44:31.563365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:31.563282    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:44:31.563365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:31.563312    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:44:31.918012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:31.917924    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" event={"ID":"e2a75412-1fd2-4c36-a3b3-4d87736d48a4","Type":"ContainerStarted","Data":"7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b"}
Oct 30 13:44:31.918012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:31.917983    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" event={"ID":"e2a75412-1fd2-4c36-a3b3-4d87736d48a4","Type":"ContainerStarted","Data":"c1029597e3e9575287135e23733b7110fce1f66231894c60203e3f15b38c2413"}
Oct 30 13:44:31.982139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:31.982066    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podStartSLOduration=1.982047565 podStartE2EDuration="1.982047565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:31.981088717 +0000 UTC m=+4336.456808329" watchObservedRunningTime="2026-10-30 13:44:31.982047565 +0000 UTC m=+4336.457767176"
Oct 30 13:44:32.213943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:32.213857    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:32.215850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:32.215816    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-f8kbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 13:44:32.216013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:32.215878    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 13:44:33.255251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:33.255187    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-f8kbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 13:44:33.256253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:33.256196    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 13:44:34.215027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:34.214986    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-f8kbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 13:44:34.215241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:34.215059    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 13:44:35.214534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:35.214498    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-f8kbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 13:44:35.215056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:35.214560    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 13:44:36.215725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:36.215681    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-f8kbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 13:44:36.216175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:36.215751    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 13:44:37.214162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:37.214119    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-f8kbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 13:44:37.214501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:37.214467    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 13:44:38.214293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:38.214252    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-f8kbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 13:44:38.214822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:38.214317    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 13:44:38.578770 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:44:38.578720    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:44:38.970222 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:38.970127    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"]
Oct 30 13:44:38.975616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:38.975584    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:38.995136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:38.995105    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:44:39.001638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.001610    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"]
Oct 30 13:44:39.007191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.007169    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:39.007346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.007169    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:44:39.013110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.013085    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:44:39.137992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.137953    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.137992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.137994    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.138375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.138345    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xn5\" (UniqueName: \"kubernetes.io/projected/cc565eb2-e27c-433a-a5a4-0e6219ea7670-kube-api-access-s8xn5\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.214948 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.214914    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-f8kbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Oct 30 13:44:39.215482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.214972    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Oct 30 13:44:39.240329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.240292    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xn5\" (UniqueName: \"kubernetes.io/projected/cc565eb2-e27c-433a-a5a4-0e6219ea7670-kube-api-access-s8xn5\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.240532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.240511    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.240602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.240544    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.241184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.241155    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.243199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.243171    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.279548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.279483    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xn5\" (UniqueName: \"kubernetes.io/projected/cc565eb2-e27c-433a-a5a4-0e6219ea7670-kube-api-access-s8xn5\") pod \"maas-api-revoke-sub-c777df28-9sxgd\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.292103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.292073    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:39.609651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:39.609621    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"]
Oct 30 13:44:39.611641 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:39.611612    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc565eb2_e27c_433a_a5a4_0e6219ea7670.slice/crio-c2320a7d0311120239eca9a549bdde69cb117951c9a781b66cd8553228fe7728 WatchSource:0}: Error finding container c2320a7d0311120239eca9a549bdde69cb117951c9a781b66cd8553228fe7728: Status 404 returned error can't find the container with id c2320a7d0311120239eca9a549bdde69cb117951c9a781b66cd8553228fe7728
Oct 30 13:44:40.068918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:40.068860    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd" event={"ID":"cc565eb2-e27c-433a-a5a4-0e6219ea7670","Type":"ContainerStarted","Data":"bd872afb2efb13b391a9bedb75daca4cf3c831a37492d03f187a20a897771bce"}
Oct 30 13:44:40.069122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:40.068938    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd" event={"ID":"cc565eb2-e27c-433a-a5a4-0e6219ea7670","Type":"ContainerStarted","Data":"c2320a7d0311120239eca9a549bdde69cb117951c9a781b66cd8553228fe7728"}
Oct 30 13:44:40.111722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:40.111650    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd" podStartSLOduration=2.111630271 podStartE2EDuration="2.111630271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:40.109318861 +0000 UTC m=+4344.585038486" watchObservedRunningTime="2026-10-30 13:44:40.111630271 +0000 UTC m=+4344.587349885"
Oct 30 13:44:40.835236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:40.824584    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:40.835236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:40.826480    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:40.835236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:40.832297    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:44:41.082939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:41.082895    4993 generic.go:356] "Generic (PLEG): container finished" podID="cc565eb2-e27c-433a-a5a4-0e6219ea7670" containerID="bd872afb2efb13b391a9bedb75daca4cf3c831a37492d03f187a20a897771bce" exitCode=0
Oct 30 13:44:41.083156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:41.082981    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd" event={"ID":"cc565eb2-e27c-433a-a5a4-0e6219ea7670","Type":"ContainerDied","Data":"bd872afb2efb13b391a9bedb75daca4cf3c831a37492d03f187a20a897771bce"}
Oct 30 13:44:42.286977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.286948    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:42.331768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.331732    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-openshift-ca\") pod \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") "
Oct 30 13:44:42.332137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.332099    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-openshift-ca" pod "cc565eb2-e27c-433a-a5a4-0e6219ea7670" (UID: "cc565eb2-e27c-433a-a5a4-0e6219ea7670"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:42.332393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.332139    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-xks-ca\") pod \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") "
Oct 30 13:44:42.332393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.332285    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc565eb2-e27c-433a-a5a4-0e6219ea7670-kube-api-access-s8xn5\" (UniqueName: \"kubernetes.io/projected/cc565eb2-e27c-433a-a5a4-0e6219ea7670-kube-api-access-s8xn5\") pod \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\" (UID: \"cc565eb2-e27c-433a-a5a4-0e6219ea7670\") "
Oct 30 13:44:42.333043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.333021    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:42.335940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.335906    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc565eb2-e27c-433a-a5a4-0e6219ea7670-kube-api-access-s8xn5" pod "cc565eb2-e27c-433a-a5a4-0e6219ea7670" (UID: "cc565eb2-e27c-433a-a5a4-0e6219ea7670"). InnerVolumeSpecName "kube-api-access-s8xn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:42.336658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.336625    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-xks-ca" pod "cc565eb2-e27c-433a-a5a4-0e6219ea7670" (UID: "cc565eb2-e27c-433a-a5a4-0e6219ea7670"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:42.434292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.434155    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc565eb2-e27c-433a-a5a4-0e6219ea7670-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:42.434292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:42.434191    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xn5\" (UniqueName: \"kubernetes.io/projected/cc565eb2-e27c-433a-a5a4-0e6219ea7670-kube-api-access-s8xn5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:43.108101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:43.108062    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd" event={"ID":"cc565eb2-e27c-433a-a5a4-0e6219ea7670","Type":"ContainerDied","Data":"c2320a7d0311120239eca9a549bdde69cb117951c9a781b66cd8553228fe7728"}
Oct 30 13:44:43.108444 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:43.108422    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2320a7d0311120239eca9a549bdde69cb117951c9a781b66cd8553228fe7728"
Oct 30 13:44:43.108574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:43.108086    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"
Oct 30 13:44:44.121747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.106471    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"]
Oct 30 13:44:44.389357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.370851    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"]
Oct 30 13:44:44.389357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.371906    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cc565eb2-e27c-433a-a5a4-0e6219ea7670" containerName="revoke-keys"
Oct 30 13:44:44.389357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.371930    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc565eb2-e27c-433a-a5a4-0e6219ea7670" containerName="revoke-keys"
Oct 30 13:44:44.389357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.372099    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc565eb2-e27c-433a-a5a4-0e6219ea7670" containerName="revoke-keys"
Oct 30 13:44:44.390087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.390046    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.409413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.409378    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"]
Oct 30 13:44:44.448018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.447979    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:44:44.448915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.448513    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:44.490961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.490919    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.491182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.490988    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.491277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.491189    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x72m\" (UniqueName: \"kubernetes.io/projected/137eb5d2-b8e8-4a7a-af89-1983f66d446c-kube-api-access-8x72m\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.495165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.495130    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:44:44.495327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.495286    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:44:44.593039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.593002    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x72m\" (UniqueName: \"kubernetes.io/projected/137eb5d2-b8e8-4a7a-af89-1983f66d446c-kube-api-access-8x72m\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.593297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.593060    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.593297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.593126    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.594110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.594072    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.595745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.595715    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.616357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.616330    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x72m\" (UniqueName: \"kubernetes.io/projected/137eb5d2-b8e8-4a7a-af89-1983f66d446c-kube-api-access-8x72m\") pod \"maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:44.769076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:44.769039    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:45.161644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.161612    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"]
Oct 30 13:44:45.162567 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:45.162533    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137eb5d2_b8e8_4a7a_af89_1983f66d446c.slice/crio-ea8a8dae5582aa2d734484d68b70e73ae2da88d08cf3247511f9db307ec05588 WatchSource:0}: Error finding container ea8a8dae5582aa2d734484d68b70e73ae2da88d08cf3247511f9db307ec05588: Status 404 returned error can't find the container with id ea8a8dae5582aa2d734484d68b70e73ae2da88d08cf3247511f9db307ec05588
Oct 30 13:44:45.800225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.800162    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"]
Oct 30 13:44:45.804764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.804736    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.843106 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.843067    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"]
Oct 30 13:44:45.915034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.914989    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d892df7c-4301-4f22-aaf8-ceca873df61b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.915259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.915045    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.915259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.915143    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.915381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.915263    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.915431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.915387    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzsr\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-kube-api-access-wbzsr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.915523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.915500    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.915581 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.915567    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.915662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.915646    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:45.915734 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:45.915719    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.016984    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzsr\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-kube-api-access-wbzsr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017083    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017143    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017194    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017302    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017429    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d892df7c-4301-4f22-aaf8-ceca873df61b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017467    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017539    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017586    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017637    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017712    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017838    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.017940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.017911    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.018163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.018145    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d892df7c-4301-4f22-aaf8-ceca873df61b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.020045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.020017    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.020301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.020282    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.040370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.040328    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.040501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.040412    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzsr\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-kube-api-access-wbzsr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rfmxm\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.125954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.125872    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:46.139618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.139574    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr" event={"ID":"137eb5d2-b8e8-4a7a-af89-1983f66d446c","Type":"ContainerStarted","Data":"8b6723c97c1bda66aaf7c7a0129ac865206b1c32371bd897bb7d954b78369856"}
Oct 30 13:44:46.139753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.139650    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr" event={"ID":"137eb5d2-b8e8-4a7a-af89-1983f66d446c","Type":"ContainerStarted","Data":"ea8a8dae5582aa2d734484d68b70e73ae2da88d08cf3247511f9db307ec05588"}
Oct 30 13:44:46.187691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.187471    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr" podStartSLOduration=2.1874457290000002 podStartE2EDuration="2.187445729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:46.181064475 +0000 UTC m=+4350.656784099" watchObservedRunningTime="2026-10-30 13:44:46.187445729 +0000 UTC m=+4350.663165345"
Oct 30 13:44:46.455244 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:46.455181    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd892df7c_4301_4f22_aaf8_ceca873df61b.slice/crio-8c2c7261ded2448959474665a4735b3dc210510862023d7585cef8b954f27d21 WatchSource:0}: Error finding container 8c2c7261ded2448959474665a4735b3dc210510862023d7585cef8b954f27d21: Status 404 returned error can't find the container with id 8c2c7261ded2448959474665a4735b3dc210510862023d7585cef8b954f27d21
Oct 30 13:44:46.455548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.455517    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"]
Oct 30 13:44:46.460715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.460188    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:44:46.460715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.460360    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:44:46.460715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:46.460453    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:44:47.149838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:47.149798    4993 generic.go:356] "Generic (PLEG): container finished" podID="137eb5d2-b8e8-4a7a-af89-1983f66d446c" containerID="8b6723c97c1bda66aaf7c7a0129ac865206b1c32371bd897bb7d954b78369856" exitCode=0
Oct 30 13:44:47.150055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:47.149881    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr" event={"ID":"137eb5d2-b8e8-4a7a-af89-1983f66d446c","Type":"ContainerDied","Data":"8b6723c97c1bda66aaf7c7a0129ac865206b1c32371bd897bb7d954b78369856"}
Oct 30 13:44:47.152144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:47.152112    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" event={"ID":"d892df7c-4301-4f22-aaf8-ceca873df61b","Type":"ContainerStarted","Data":"8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e"}
Oct 30 13:44:47.152316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:47.152159    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" event={"ID":"d892df7c-4301-4f22-aaf8-ceca873df61b","Type":"ContainerStarted","Data":"8c2c7261ded2448959474665a4735b3dc210510862023d7585cef8b954f27d21"}
Oct 30 13:44:47.244501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:47.244441    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podStartSLOduration=2.244423207 podStartE2EDuration="2.244423207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:47.243579141 +0000 UTC m=+4351.719298752" watchObservedRunningTime="2026-10-30 13:44:47.244423207 +0000 UTC m=+4351.720142819"
Oct 30 13:44:48.126970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.126933    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:48.129135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.129099    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rfmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:44:48.129318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.129176    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 13:44:48.524517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.524488    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:48.621655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.621617    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-openshift-ca\") pod \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") "
Oct 30 13:44:48.622188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.621962    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-xks-ca\") pod \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") "
Oct 30 13:44:48.622188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.622096    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-openshift-ca" pod "137eb5d2-b8e8-4a7a-af89-1983f66d446c" (UID: "137eb5d2-b8e8-4a7a-af89-1983f66d446c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:48.622188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.622136    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137eb5d2-b8e8-4a7a-af89-1983f66d446c-kube-api-access-8x72m\" (UniqueName: \"kubernetes.io/projected/137eb5d2-b8e8-4a7a-af89-1983f66d446c-kube-api-access-8x72m\") pod \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\" (UID: \"137eb5d2-b8e8-4a7a-af89-1983f66d446c\") "
Oct 30 13:44:48.623172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.623119    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.626034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.626009    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137eb5d2-b8e8-4a7a-af89-1983f66d446c-kube-api-access-8x72m" pod "137eb5d2-b8e8-4a7a-af89-1983f66d446c" (UID: "137eb5d2-b8e8-4a7a-af89-1983f66d446c"). InnerVolumeSpecName "kube-api-access-8x72m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:48.626373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.626349    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-xks-ca" pod "137eb5d2-b8e8-4a7a-af89-1983f66d446c" (UID: "137eb5d2-b8e8-4a7a-af89-1983f66d446c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:48.724308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.724265    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137eb5d2-b8e8-4a7a-af89-1983f66d446c-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.724308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:48.724302    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x72m\" (UniqueName: \"kubernetes.io/projected/137eb5d2-b8e8-4a7a-af89-1983f66d446c-kube-api-access-8x72m\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:49.127074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:49.127032    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rfmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:44:49.127273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:49.127097    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 13:44:49.198968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:49.198854    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"
Oct 30 13:44:49.198968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:49.198887    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr" event={"ID":"137eb5d2-b8e8-4a7a-af89-1983f66d446c","Type":"ContainerDied","Data":"ea8a8dae5582aa2d734484d68b70e73ae2da88d08cf3247511f9db307ec05588"}
Oct 30 13:44:49.198968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:49.198941    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea8a8dae5582aa2d734484d68b70e73ae2da88d08cf3247511f9db307ec05588"
Oct 30 13:44:50.126696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:50.126658    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rfmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:44:50.127099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:50.126724    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 13:44:51.127049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.127013    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rfmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:44:51.127497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.127067    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 13:44:51.286613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.286565    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"]
Oct 30 13:44:51.287540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.287516    4993 cpu_manager.go:414] "containerMap: removing container" podUID="137eb5d2-b8e8-4a7a-af89-1983f66d446c" containerName="revoke-keys"
Oct 30 13:44:51.287540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.287537    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137eb5d2-b8e8-4a7a-af89-1983f66d446c" containerName="revoke-keys"
Oct 30 13:44:51.287750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.287660    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="137eb5d2-b8e8-4a7a-af89-1983f66d446c" containerName="revoke-keys"
Oct 30 13:44:51.291309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.291292    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.307434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.307376    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"]
Oct 30 13:44:51.307698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.307397    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:44:51.313483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.313454    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:51.313483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.313454    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:44:51.319261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.319192    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:44:51.392341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.390041    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.392341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.390197    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.392341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.390264    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jtxw\" (UniqueName: \"kubernetes.io/projected/ced2627f-1550-49aa-b455-ac3581ef9ef6-kube-api-access-9jtxw\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.492745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.492687    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.492941 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.492823    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.492941 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.492875    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtxw\" (UniqueName: \"kubernetes.io/projected/ced2627f-1550-49aa-b455-ac3581ef9ef6-kube-api-access-9jtxw\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.493692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.493659    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.495738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.495714    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.516880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.516841    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtxw\" (UniqueName: \"kubernetes.io/projected/ced2627f-1550-49aa-b455-ac3581ef9ef6-kube-api-access-9jtxw\") pod \"maas-api-revoke-sub-efc1258a-sv2w6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.607859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.607817    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:51.986742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:51.986095    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"]
Oct 30 13:44:51.988299 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:51.988242    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced2627f_1550_49aa_b455_ac3581ef9ef6.slice/crio-c8f8bcd44e53b8c3765c67dff1fe7f8744803ed8fbeaac2b7c3c874ad93f8067 WatchSource:0}: Error finding container c8f8bcd44e53b8c3765c67dff1fe7f8744803ed8fbeaac2b7c3c874ad93f8067: Status 404 returned error can't find the container with id c8f8bcd44e53b8c3765c67dff1fe7f8744803ed8fbeaac2b7c3c874ad93f8067
Oct 30 13:44:52.129640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:52.126123    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rfmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:44:52.129640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:52.126173    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 13:44:52.230721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:52.230683    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6" event={"ID":"ced2627f-1550-49aa-b455-ac3581ef9ef6","Type":"ContainerStarted","Data":"c8f8bcd44e53b8c3765c67dff1fe7f8744803ed8fbeaac2b7c3c874ad93f8067"}
Oct 30 13:44:52.884312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:52.884197    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"]
Oct 30 13:44:52.888806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:52.888778    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:52.919119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:52.919079    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"]
Oct 30 13:44:53.019964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.019919    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.019964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.019976    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.020363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.020340    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsclw\" (UniqueName: \"kubernetes.io/projected/b3537761-041e-4371-9f16-546ceceae4d6-kube-api-access-fsclw\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.122091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.122037    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsclw\" (UniqueName: \"kubernetes.io/projected/b3537761-041e-4371-9f16-546ceceae4d6-kube-api-access-fsclw\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.122323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.122297    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.122409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.122334    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.123246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.123196    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.125507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.125475    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.126473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.126447    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rfmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:44:53.126555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.126505    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 13:44:53.148084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.147998    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsclw\" (UniqueName: \"kubernetes.io/projected/b3537761-041e-4371-9f16-546ceceae4d6-kube-api-access-fsclw\") pod \"maas-api-revoke-sub-c4cce218-rbn7d\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.210926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.210884    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:53.243651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.243614    4993 generic.go:356] "Generic (PLEG): container finished" podID="ced2627f-1550-49aa-b455-ac3581ef9ef6" containerID="dadf5f79a462a9103b3e07eaede38382deff533dd8230a398a8d95f354450102" exitCode=0
Oct 30 13:44:53.243884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.243683    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6" event={"ID":"ced2627f-1550-49aa-b455-ac3581ef9ef6","Type":"ContainerDied","Data":"dadf5f79a462a9103b3e07eaede38382deff533dd8230a398a8d95f354450102"}
Oct 30 13:44:53.534351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:53.534320    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"]
Oct 30 13:44:53.535621 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:44:53.535595    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3537761_041e_4371_9f16_546ceceae4d6.slice/crio-b7f82aa11d3091be66948cbc4f5ff28af1258d4f5e273a71c5c373e591876a44 WatchSource:0}: Error finding container b7f82aa11d3091be66948cbc4f5ff28af1258d4f5e273a71c5c373e591876a44: Status 404 returned error can't find the container with id b7f82aa11d3091be66948cbc4f5ff28af1258d4f5e273a71c5c373e591876a44
Oct 30 13:44:54.126684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.126646    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rfmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:44:54.126885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.126717    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Oct 30 13:44:54.256956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.256913    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d" event={"ID":"b3537761-041e-4371-9f16-546ceceae4d6","Type":"ContainerStarted","Data":"56971980bd0e30b95e6377dae28adb92e4de6d2c047d37d54477efeed6ae0551"}
Oct 30 13:44:54.257449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.256983    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d" event={"ID":"b3537761-041e-4371-9f16-546ceceae4d6","Type":"ContainerStarted","Data":"b7f82aa11d3091be66948cbc4f5ff28af1258d4f5e273a71c5c373e591876a44"}
Oct 30 13:44:54.305403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.305340    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d" podStartSLOduration=2.30532061 podStartE2EDuration="2.30532061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:54.297892455 +0000 UTC m=+4358.773612077" watchObservedRunningTime="2026-10-30 13:44:54.30532061 +0000 UTC m=+4358.781040225"
Oct 30 13:44:54.468979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.468937    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:54.652821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.652725    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced2627f-1550-49aa-b455-ac3581ef9ef6-kube-api-access-9jtxw\" (UniqueName: \"kubernetes.io/projected/ced2627f-1550-49aa-b455-ac3581ef9ef6-kube-api-access-9jtxw\") pod \"ced2627f-1550-49aa-b455-ac3581ef9ef6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") "
Oct 30 13:44:54.652999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.652928    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-openshift-ca\") pod \"ced2627f-1550-49aa-b455-ac3581ef9ef6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") "
Oct 30 13:44:54.653324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.653295    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-xks-ca\") pod \"ced2627f-1550-49aa-b455-ac3581ef9ef6\" (UID: \"ced2627f-1550-49aa-b455-ac3581ef9ef6\") "
Oct 30 13:44:54.654479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.654446    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-openshift-ca" pod "ced2627f-1550-49aa-b455-ac3581ef9ef6" (UID: "ced2627f-1550-49aa-b455-ac3581ef9ef6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:54.657233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.657155    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-xks-ca" pod "ced2627f-1550-49aa-b455-ac3581ef9ef6" (UID: "ced2627f-1550-49aa-b455-ac3581ef9ef6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:54.657384 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.657357    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced2627f-1550-49aa-b455-ac3581ef9ef6-kube-api-access-9jtxw" pod "ced2627f-1550-49aa-b455-ac3581ef9ef6" (UID: "ced2627f-1550-49aa-b455-ac3581ef9ef6"). InnerVolumeSpecName "kube-api-access-9jtxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:54.755932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.755896    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:54.755932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.755929    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced2627f-1550-49aa-b455-ac3581ef9ef6-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:54.755932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:54.755940    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jtxw\" (UniqueName: \"kubernetes.io/projected/ced2627f-1550-49aa-b455-ac3581ef9ef6-kube-api-access-9jtxw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.270325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:55.270280    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6" event={"ID":"ced2627f-1550-49aa-b455-ac3581ef9ef6","Type":"ContainerDied","Data":"c8f8bcd44e53b8c3765c67dff1fe7f8744803ed8fbeaac2b7c3c874ad93f8067"}
Oct 30 13:44:55.271129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:55.270354    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8f8bcd44e53b8c3765c67dff1fe7f8744803ed8fbeaac2b7c3c874ad93f8067"
Oct 30 13:44:55.271129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:55.270424    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"
Oct 30 13:44:55.274825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:55.274787    4993 generic.go:356] "Generic (PLEG): container finished" podID="b3537761-041e-4371-9f16-546ceceae4d6" containerID="56971980bd0e30b95e6377dae28adb92e4de6d2c047d37d54477efeed6ae0551" exitCode=0
Oct 30 13:44:55.275045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:55.275021    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d" event={"ID":"b3537761-041e-4371-9f16-546ceceae4d6","Type":"ContainerDied","Data":"56971980bd0e30b95e6377dae28adb92e4de6d2c047d37d54477efeed6ae0551"}
Oct 30 13:44:55.492572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:55.492532    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:55.494924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:55.494882    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:55.516005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:55.515971    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:44:56.473408 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.473377    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:56.552884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.552778    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3537761-041e-4371-9f16-546ceceae4d6-kube-api-access-fsclw\" (UniqueName: \"kubernetes.io/projected/b3537761-041e-4371-9f16-546ceceae4d6-kube-api-access-fsclw\") pod \"b3537761-041e-4371-9f16-546ceceae4d6\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") "
Oct 30 13:44:56.553078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.553017    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-xks-ca\") pod \"b3537761-041e-4371-9f16-546ceceae4d6\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") "
Oct 30 13:44:56.553078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.553069    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-openshift-ca\") pod \"b3537761-041e-4371-9f16-546ceceae4d6\" (UID: \"b3537761-041e-4371-9f16-546ceceae4d6\") "
Oct 30 13:44:56.553680 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.553652    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-openshift-ca" pod "b3537761-041e-4371-9f16-546ceceae4d6" (UID: "b3537761-041e-4371-9f16-546ceceae4d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:56.555786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.555753    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-xks-ca" pod "b3537761-041e-4371-9f16-546ceceae4d6" (UID: "b3537761-041e-4371-9f16-546ceceae4d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:56.559416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.559377    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3537761-041e-4371-9f16-546ceceae4d6-kube-api-access-fsclw" pod "b3537761-041e-4371-9f16-546ceceae4d6" (UID: "b3537761-041e-4371-9f16-546ceceae4d6"). InnerVolumeSpecName "kube-api-access-fsclw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:56.655009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.654968    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsclw\" (UniqueName: \"kubernetes.io/projected/b3537761-041e-4371-9f16-546ceceae4d6-kube-api-access-fsclw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:56.655009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.655005    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:56.655009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:56.655014    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3537761-041e-4371-9f16-546ceceae4d6-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:57.314380 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:57.314341    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"
Oct 30 13:44:57.314580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:57.314376    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d" event={"ID":"b3537761-041e-4371-9f16-546ceceae4d6","Type":"ContainerDied","Data":"b7f82aa11d3091be66948cbc4f5ff28af1258d4f5e273a71c5c373e591876a44"}
Oct 30 13:44:57.314580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:57.314441    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7f82aa11d3091be66948cbc4f5ff28af1258d4f5e273a71c5c373e591876a44"
Oct 30 13:44:59.154652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.154601    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p" podUID="b96f62f9-a6d6-459d-a3f3-59ea00bafaac" containerName="main" containerID="cri-o://9fb90b29d4d72f30583dd5abbbc9b395a04f4a93b6845dc3bb7d46effa7bd05a" gracePeriod=45
Oct 30 13:44:59.335661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.335624    4993 generic.go:356] "Generic (PLEG): container finished" podID="b96f62f9-a6d6-459d-a3f3-59ea00bafaac" containerID="9fb90b29d4d72f30583dd5abbbc9b395a04f4a93b6845dc3bb7d46effa7bd05a" exitCode=0
Oct 30 13:44:59.335859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.335703    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p" event={"ID":"b96f62f9-a6d6-459d-a3f3-59ea00bafaac","Type":"ContainerDied","Data":"9fb90b29d4d72f30583dd5abbbc9b395a04f4a93b6845dc3bb7d46effa7bd05a"}
Oct 30 13:44:59.464225 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.464185    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:44:59.504737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.504697    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tls-certs\" (UniqueName: \"kubernetes.io/secret/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tls-certs\") pod \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") "
Oct 30 13:44:59.505001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.504759    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-model-cache\") pod \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") "
Oct 30 13:44:59.505001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.504850    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-home\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-home\") pod \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") "
Oct 30 13:44:59.505001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.504951    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-kube-api-access-8cz2v\" (UniqueName: \"kubernetes.io/projected/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-kube-api-access-8cz2v\") pod \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") "
Oct 30 13:44:59.505366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.505007    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tmp-dir\") pod \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") "
Oct 30 13:44:59.505366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.505102    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-model-cache" pod "b96f62f9-a6d6-459d-a3f3-59ea00bafaac" (UID: "b96f62f9-a6d6-459d-a3f3-59ea00bafaac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:59.505366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.505199    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-home" pod "b96f62f9-a6d6-459d-a3f3-59ea00bafaac" (UID: "b96f62f9-a6d6-459d-a3f3-59ea00bafaac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:59.505366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.505288    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-dshm\") pod \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\" (UID: \"b96f62f9-a6d6-459d-a3f3-59ea00bafaac\") "
Oct 30 13:44:59.505584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.505507    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tmp-dir" pod "b96f62f9-a6d6-459d-a3f3-59ea00bafaac" (UID: "b96f62f9-a6d6-459d-a3f3-59ea00bafaac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:59.506034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.505989    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:59.506034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.506015    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:59.506034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.506027    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:59.508086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.508055    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tls-certs" pod "b96f62f9-a6d6-459d-a3f3-59ea00bafaac" (UID: "b96f62f9-a6d6-459d-a3f3-59ea00bafaac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:59.508410 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.508384    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-dshm" pod "b96f62f9-a6d6-459d-a3f3-59ea00bafaac" (UID: "b96f62f9-a6d6-459d-a3f3-59ea00bafaac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:59.513241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.508592    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-kube-api-access-8cz2v" pod "b96f62f9-a6d6-459d-a3f3-59ea00bafaac" (UID: "b96f62f9-a6d6-459d-a3f3-59ea00bafaac"). InnerVolumeSpecName "kube-api-access-8cz2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:59.607731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.607694    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:59.607731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.607727    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cz2v\" (UniqueName: \"kubernetes.io/projected/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-kube-api-access-8cz2v\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:44:59.607731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:44:59.607738    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96f62f9-a6d6-459d-a3f3-59ea00bafaac-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:00.143634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.143596    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"]
Oct 30 13:45:00.144239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144223    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ced2627f-1550-49aa-b455-ac3581ef9ef6" containerName="revoke-keys"
Oct 30 13:45:00.144239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144240    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced2627f-1550-49aa-b455-ac3581ef9ef6" containerName="revoke-keys"
Oct 30 13:45:00.144359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144254    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b96f62f9-a6d6-459d-a3f3-59ea00bafaac" containerName="main"
Oct 30 13:45:00.144359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144259    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96f62f9-a6d6-459d-a3f3-59ea00bafaac" containerName="main"
Oct 30 13:45:00.144359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144307    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b3537761-041e-4371-9f16-546ceceae4d6" containerName="revoke-keys"
Oct 30 13:45:00.144359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144315    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3537761-041e-4371-9f16-546ceceae4d6" containerName="revoke-keys"
Oct 30 13:45:00.144540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144489    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3537761-041e-4371-9f16-546ceceae4d6" containerName="revoke-keys"
Oct 30 13:45:00.144540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144509    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced2627f-1550-49aa-b455-ac3581ef9ef6" containerName="revoke-keys"
Oct 30 13:45:00.144540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.144519    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96f62f9-a6d6-459d-a3f3-59ea00bafaac" containerName="main"
Oct 30 13:45:00.148483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.148459    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:00.150773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.150741    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"]
Oct 30 13:45:00.155164 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.155143    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:00.160974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.160952    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"]
Oct 30 13:45:00.166336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.166313    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"]
Oct 30 13:45:00.166489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.166462    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:00.171636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.171613    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"]
Oct 30 13:45:00.171762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.171744    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:00.172950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.172926    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:45:00.173085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.172996    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:45:00.175987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.175957    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:00.177495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.177474    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:45:00.191504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.187703    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"]
Oct 30 13:45:00.194769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.194499    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:00.198912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.198880    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"]
Oct 30 13:45:00.214139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.208296    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"]
Oct 30 13:45:00.218087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.218050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbpn7\" (UniqueName: \"kubernetes.io/projected/613f276e-cad4-40e0-84e8-b5c7e2524853-kube-api-access-mbpn7\") pod \"maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj\" (UID: \"613f276e-cad4-40e0-84e8-b5c7e2524853\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:00.218376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.218348    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8pd5\" (UniqueName: \"kubernetes.io/projected/b96952ed-4208-4568-9005-6a53e3f36eb6-kube-api-access-v8pd5\") pod \"maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8\" (UID: \"b96952ed-4208-4568-9005-6a53e3f36eb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:00.218558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.218538    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96952ed-4208-4568-9005-6a53e3f36eb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8\" (UID: \"b96952ed-4208-4568-9005-6a53e3f36eb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:00.218731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.218713    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6t2f\" (UniqueName: \"kubernetes.io/projected/48a48c09-e2e1-4006-8d0c-2331997f7a85-kube-api-access-l6t2f\") pod \"maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4\" (UID: \"48a48c09-e2e1-4006-8d0c-2331997f7a85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:00.218950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.218911    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw\" (UID: \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:00.219144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.219126    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0393d0-1fa2-4248-9d19-f31b08556afa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8\" (UID: \"7f0393d0-1fa2-4248-9d19-f31b08556afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:00.219353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.219334    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k56z\" (UniqueName: \"kubernetes.io/projected/7f0393d0-1fa2-4248-9d19-f31b08556afa-kube-api-access-4k56z\") pod \"maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8\" (UID: \"7f0393d0-1fa2-4248-9d19-f31b08556afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:00.219518 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.219500    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrtw\" (UniqueName: \"kubernetes.io/projected/de1bfff9-9805-4b5a-a959-734a9a1570e8-kube-api-access-xkrtw\") pod \"maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm\" (UID: \"de1bfff9-9805-4b5a-a959-734a9a1570e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:00.219649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.219632    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmkw6\" (UniqueName: \"kubernetes.io/projected/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-kube-api-access-mmkw6\") pod \"maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw\" (UID: \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:00.219819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.219800    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1bfff9-9805-4b5a-a959-734a9a1570e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm\" (UID: \"de1bfff9-9805-4b5a-a959-734a9a1570e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:00.220008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.219991    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613f276e-cad4-40e0-84e8-b5c7e2524853-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj\" (UID: \"613f276e-cad4-40e0-84e8-b5c7e2524853\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:00.220134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.220120    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a48c09-e2e1-4006-8d0c-2331997f7a85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4\" (UID: \"48a48c09-e2e1-4006-8d0c-2331997f7a85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:00.226970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.226765    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"]
Oct 30 13:45:00.234746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.234251    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"]
Oct 30 13:45:00.249286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.249253    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:00.251358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.251329    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"]
Oct 30 13:45:00.260631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.260439    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:00.265466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.264167    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"]
Oct 30 13:45:00.319766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.319728    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:00.322281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.322241    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0393d0-1fa2-4248-9d19-f31b08556afa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8\" (UID: \"7f0393d0-1fa2-4248-9d19-f31b08556afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:00.324928 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.324684    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k56z\" (UniqueName: \"kubernetes.io/projected/7f0393d0-1fa2-4248-9d19-f31b08556afa-kube-api-access-4k56z\") pod \"maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8\" (UID: \"7f0393d0-1fa2-4248-9d19-f31b08556afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:00.325135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.325111    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrtw\" (UniqueName: \"kubernetes.io/projected/de1bfff9-9805-4b5a-a959-734a9a1570e8-kube-api-access-xkrtw\") pod \"maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm\" (UID: \"de1bfff9-9805-4b5a-a959-734a9a1570e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:00.325298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.325281    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmkw6\" (UniqueName: \"kubernetes.io/projected/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-kube-api-access-mmkw6\") pod \"maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw\" (UID: \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:00.325492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.325473    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1bfff9-9805-4b5a-a959-734a9a1570e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm\" (UID: \"de1bfff9-9805-4b5a-a959-734a9a1570e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:00.325646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.325627    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782\" (UID: \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:00.330990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.324965    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"]
Oct 30 13:45:00.330990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.323033    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0393d0-1fa2-4248-9d19-f31b08556afa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8\" (UID: \"7f0393d0-1fa2-4248-9d19-f31b08556afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:00.330990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.326450    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1bfff9-9805-4b5a-a959-734a9a1570e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm\" (UID: \"de1bfff9-9805-4b5a-a959-734a9a1570e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:00.336605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.336574    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldtw9\" (UniqueName: \"kubernetes.io/projected/380b156b-fe53-460f-aa79-2a9f98e37bd2-kube-api-access-ldtw9\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm\" (UID: \"380b156b-fe53-460f-aa79-2a9f98e37bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:00.342241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.342174    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613f276e-cad4-40e0-84e8-b5c7e2524853-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj\" (UID: \"613f276e-cad4-40e0-84e8-b5c7e2524853\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:00.342567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.342539    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a48c09-e2e1-4006-8d0c-2331997f7a85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4\" (UID: \"48a48c09-e2e1-4006-8d0c-2331997f7a85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:00.343267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.343244    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbpn7\" (UniqueName: \"kubernetes.io/projected/613f276e-cad4-40e0-84e8-b5c7e2524853-kube-api-access-mbpn7\") pod \"maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj\" (UID: \"613f276e-cad4-40e0-84e8-b5c7e2524853\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:00.343779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.343757    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8pd5\" (UniqueName: \"kubernetes.io/projected/b96952ed-4208-4568-9005-6a53e3f36eb6-kube-api-access-v8pd5\") pod \"maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8\" (UID: \"b96952ed-4208-4568-9005-6a53e3f36eb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:00.345066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.345042    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96952ed-4208-4568-9005-6a53e3f36eb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8\" (UID: \"b96952ed-4208-4568-9005-6a53e3f36eb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:00.345505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.345482    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b156b-fe53-460f-aa79-2a9f98e37bd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm\" (UID: \"380b156b-fe53-460f-aa79-2a9f98e37bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:00.346485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.346464    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6t2f\" (UniqueName: \"kubernetes.io/projected/48a48c09-e2e1-4006-8d0c-2331997f7a85-kube-api-access-l6t2f\") pod \"maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4\" (UID: \"48a48c09-e2e1-4006-8d0c-2331997f7a85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:00.346947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.346650    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpft\" (UniqueName: \"kubernetes.io/projected/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-kube-api-access-8qpft\") pod \"maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782\" (UID: \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:00.349161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.349140    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw\" (UID: \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:00.350238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.350198    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw\" (UID: \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:00.350376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.343269    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613f276e-cad4-40e0-84e8-b5c7e2524853-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj\" (UID: \"613f276e-cad4-40e0-84e8-b5c7e2524853\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:00.351504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.351471    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96952ed-4208-4568-9005-6a53e3f36eb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8\" (UID: \"b96952ed-4208-4568-9005-6a53e3f36eb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:00.351675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.346354    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a48c09-e2e1-4006-8d0c-2331997f7a85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4\" (UID: \"48a48c09-e2e1-4006-8d0c-2331997f7a85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:00.386064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.370631    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k56z\" (UniqueName: \"kubernetes.io/projected/7f0393d0-1fa2-4248-9d19-f31b08556afa-kube-api-access-4k56z\") pod \"maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8\" (UID: \"7f0393d0-1fa2-4248-9d19-f31b08556afa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:00.386064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.385483    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrtw\" (UniqueName: \"kubernetes.io/projected/de1bfff9-9805-4b5a-a959-734a9a1570e8-kube-api-access-xkrtw\") pod \"maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm\" (UID: \"de1bfff9-9805-4b5a-a959-734a9a1570e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:00.405635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.404824    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p" event={"ID":"b96f62f9-a6d6-459d-a3f3-59ea00bafaac","Type":"ContainerDied","Data":"7443090734ec3ee49b87437e4e1aad64e3f6ee04eb294e2f82d27e859c2793c3"}
Oct 30 13:45:00.405635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.404916    4993 scope.go:122] "RemoveContainer" containerID="9fb90b29d4d72f30583dd5abbbc9b395a04f4a93b6845dc3bb7d46effa7bd05a"
Oct 30 13:45:00.405635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.405191    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"
Oct 30 13:45:00.425400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.425363    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"]
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.443877    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8pd5\" (UniqueName: \"kubernetes.io/projected/b96952ed-4208-4568-9005-6a53e3f36eb6-kube-api-access-v8pd5\") pod \"maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8\" (UID: \"b96952ed-4208-4568-9005-6a53e3f36eb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.444591    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmkw6\" (UniqueName: \"kubernetes.io/projected/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-kube-api-access-mmkw6\") pod \"maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw\" (UID: \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.448616    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.449756    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6t2f\" (UniqueName: \"kubernetes.io/projected/48a48c09-e2e1-4006-8d0c-2331997f7a85-kube-api-access-l6t2f\") pod \"maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4\" (UID: \"48a48c09-e2e1-4006-8d0c-2331997f7a85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.451332    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbpn7\" (UniqueName: \"kubernetes.io/projected/613f276e-cad4-40e0-84e8-b5c7e2524853-kube-api-access-mbpn7\") pod \"maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj\" (UID: \"613f276e-cad4-40e0-84e8-b5c7e2524853\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.451767    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpft\" (UniqueName: \"kubernetes.io/projected/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-kube-api-access-8qpft\") pod \"maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782\" (UID: \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.451802    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8km\" (UniqueName: \"kubernetes.io/projected/2907709a-ad1d-4724-ab42-02ad7defb4b1-kube-api-access-6t8km\") pod \"maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8\" (UID: \"2907709a-ad1d-4724-ab42-02ad7defb4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.451983    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2907709a-ad1d-4724-ab42-02ad7defb4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8\" (UID: \"2907709a-ad1d-4724-ab42-02ad7defb4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.452038    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782\" (UID: \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.452097    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldtw9\" (UniqueName: \"kubernetes.io/projected/380b156b-fe53-460f-aa79-2a9f98e37bd2-kube-api-access-ldtw9\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm\" (UID: \"380b156b-fe53-460f-aa79-2a9f98e37bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.452338    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b156b-fe53-460f-aa79-2a9f98e37bd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm\" (UID: \"380b156b-fe53-460f-aa79-2a9f98e37bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.453110    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b156b-fe53-460f-aa79-2a9f98e37bd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm\" (UID: \"380b156b-fe53-460f-aa79-2a9f98e37bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.454239    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782\" (UID: \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:00.445151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.454512    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"]
Oct 30 13:45:00.468470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.468279    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"]
Oct 30 13:45:00.475283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.472843    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:00.477414 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.477335    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:00.478475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.478450    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:00.488338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.488254    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"]
Oct 30 13:45:00.500596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.496628    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:00.500596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.498897    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"]
Oct 30 13:45:00.504345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.503456    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpft\" (UniqueName: \"kubernetes.io/projected/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-kube-api-access-8qpft\") pod \"maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782\" (UID: \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:00.513877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.513844    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldtw9\" (UniqueName: \"kubernetes.io/projected/380b156b-fe53-460f-aa79-2a9f98e37bd2-kube-api-access-ldtw9\") pod \"maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm\" (UID: \"380b156b-fe53-460f-aa79-2a9f98e37bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:00.515498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.515471    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"]
Oct 30 13:45:00.531199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.528286    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"]
Oct 30 13:45:00.534052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.534025    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:00.540711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.540681    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"]
Oct 30 13:45:00.549743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.549716    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"]
Oct 30 13:45:00.550561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.549958    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:00.553558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.553530    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8km\" (UniqueName: \"kubernetes.io/projected/2907709a-ad1d-4724-ab42-02ad7defb4b1-kube-api-access-6t8km\") pod \"maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8\" (UID: \"2907709a-ad1d-4724-ab42-02ad7defb4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:00.555897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.553686    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2907709a-ad1d-4724-ab42-02ad7defb4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8\" (UID: \"2907709a-ad1d-4724-ab42-02ad7defb4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:00.555897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.553731    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tq7\" (UniqueName: \"kubernetes.io/projected/998580a2-a1bf-4eb6-b8db-b3277af46957-kube-api-access-r8tq7\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9\" (UID: \"998580a2-a1bf-4eb6-b8db-b3277af46957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:00.555897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.553777    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75087b72-ac15-4d81-bd96-659ac0f7b2ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc\" (UID: \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:00.555897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.553934    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q54nt\" (UniqueName: \"kubernetes.io/projected/75087b72-ac15-4d81-bd96-659ac0f7b2ef-kube-api-access-q54nt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc\" (UID: \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:00.555897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.553994    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998580a2-a1bf-4eb6-b8db-b3277af46957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9\" (UID: \"998580a2-a1bf-4eb6-b8db-b3277af46957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:00.557265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.557002    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"]
Oct 30 13:45:00.557265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.557116    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:00.557698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.557646    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2907709a-ad1d-4724-ab42-02ad7defb4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8\" (UID: \"2907709a-ad1d-4724-ab42-02ad7defb4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:00.558897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.558801    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"]
Oct 30 13:45:00.569528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.565929    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"]
Oct 30 13:45:00.572030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.571871    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:00.582336 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.578578    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"]
Oct 30 13:45:00.583439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.583324    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:00.587709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.586483    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:00.587709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.587130    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"]
Oct 30 13:45:00.588540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.588440    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8km\" (UniqueName: \"kubernetes.io/projected/2907709a-ad1d-4724-ab42-02ad7defb4b1-kube-api-access-6t8km\") pod \"maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8\" (UID: \"2907709a-ad1d-4724-ab42-02ad7defb4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:00.600649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.600534    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:00.616008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.615891    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"]
Oct 30 13:45:00.627843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.627807    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"]
Oct 30 13:45:00.630799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.630767    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:00.660449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.660380    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"]
Oct 30 13:45:00.662556 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.662533    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:00.663125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.663105    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.668744    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589327-9884-4386-a522-df5b12d23094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg\" (UID: \"49589327-9884-4386-a522-df5b12d23094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.668803    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcqlb\" (UniqueName: \"kubernetes.io/projected/49589327-9884-4386-a522-df5b12d23094-kube-api-access-gcqlb\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg\" (UID: \"49589327-9884-4386-a522-df5b12d23094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.668850    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tq7\" (UniqueName: \"kubernetes.io/projected/998580a2-a1bf-4eb6-b8db-b3277af46957-kube-api-access-r8tq7\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9\" (UID: \"998580a2-a1bf-4eb6-b8db-b3277af46957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.668901    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75087b72-ac15-4d81-bd96-659ac0f7b2ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc\" (UID: \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.668931    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27\" (UID: \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.669031    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e7cb2-282d-4f7e-b4ca-952b9df7a041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg\" (UID: \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.669120    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q54nt\" (UniqueName: \"kubernetes.io/projected/75087b72-ac15-4d81-bd96-659ac0f7b2ef-kube-api-access-q54nt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc\" (UID: \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.669166    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66456109-7df3-4de7-978e-13496801fff2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h\" (UID: \"66456109-7df3-4de7-978e-13496801fff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:00.686721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.669957    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75087b72-ac15-4d81-bd96-659ac0f7b2ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc\" (UID: \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:00.689352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.669201    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgx5z\" (UniqueName: \"kubernetes.io/projected/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-kube-api-access-jgx5z\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27\" (UID: \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:00.689352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.687379    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998580a2-a1bf-4eb6-b8db-b3277af46957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9\" (UID: \"998580a2-a1bf-4eb6-b8db-b3277af46957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:00.689352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.687421    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2wp\" (UniqueName: \"kubernetes.io/projected/66456109-7df3-4de7-978e-13496801fff2-kube-api-access-xn2wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h\" (UID: \"66456109-7df3-4de7-978e-13496801fff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:00.689352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.687558    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv696\" (UniqueName: \"kubernetes.io/projected/552e7cb2-282d-4f7e-b4ca-952b9df7a041-kube-api-access-wv696\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg\" (UID: \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:00.689352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.687669    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd449\" (UniqueName: \"kubernetes.io/projected/816d95a5-15ed-418c-93f1-79b39600a5e3-kube-api-access-jd449\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh\" (UID: \"816d95a5-15ed-418c-93f1-79b39600a5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:00.689352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.687746    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d95a5-15ed-418c-93f1-79b39600a5e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh\" (UID: \"816d95a5-15ed-418c-93f1-79b39600a5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:00.689352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.688652    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998580a2-a1bf-4eb6-b8db-b3277af46957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9\" (UID: \"998580a2-a1bf-4eb6-b8db-b3277af46957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:00.696634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.696606    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"]
Oct 30 13:45:00.696918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.696903    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"]
Oct 30 13:45:00.697105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.696943    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:00.706338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.705870    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:00.710739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.709840    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q54nt\" (UniqueName: \"kubernetes.io/projected/75087b72-ac15-4d81-bd96-659ac0f7b2ef-kube-api-access-q54nt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc\" (UID: \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:00.713730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.713647    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tq7\" (UniqueName: \"kubernetes.io/projected/998580a2-a1bf-4eb6-b8db-b3277af46957-kube-api-access-r8tq7\") pod \"maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9\" (UID: \"998580a2-a1bf-4eb6-b8db-b3277af46957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:00.722485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.714998    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"]
Oct 30 13:45:00.722485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.715045    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"]
Oct 30 13:45:00.722485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.715073    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"]
Oct 30 13:45:00.722485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.715100    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"]
Oct 30 13:45:00.722485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.720138    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:00.884434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.876995    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:00.884434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.878503    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"]
Oct 30 13:45:00.884434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.884326    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ac7f13-0f86-43e3-9189-d89aa39defb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp\" (UID: \"59ac7f13-0f86-43e3-9189-d89aa39defb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:00.884434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.884423    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589327-9884-4386-a522-df5b12d23094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg\" (UID: \"49589327-9884-4386-a522-df5b12d23094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:00.884786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.884494    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcqlb\" (UniqueName: \"kubernetes.io/projected/49589327-9884-4386-a522-df5b12d23094-kube-api-access-gcqlb\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg\" (UID: \"49589327-9884-4386-a522-df5b12d23094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:00.884786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.884558    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27\" (UID: \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:00.884786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.884684    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e7cb2-282d-4f7e-b4ca-952b9df7a041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg\" (UID: \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:00.884955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.884832    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzjs\" (UniqueName: \"kubernetes.io/projected/59ac7f13-0f86-43e3-9189-d89aa39defb9-kube-api-access-qkzjs\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp\" (UID: \"59ac7f13-0f86-43e3-9189-d89aa39defb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:00.884955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.884919    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpv4f\" (UniqueName: \"kubernetes.io/projected/a7bc62a2-0765-497f-8529-8a11acfeb6be-kube-api-access-lpv4f\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb\" (UID: \"a7bc62a2-0765-497f-8529-8a11acfeb6be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:00.885067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.884989    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf2c6\" (UniqueName: \"kubernetes.io/projected/9325a541-f5aa-4272-bbe1-c75fd95e384a-kube-api-access-jf2c6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8\" (UID: \"9325a541-f5aa-4272-bbe1-c75fd95e384a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:00.885123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.885086    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66456109-7df3-4de7-978e-13496801fff2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h\" (UID: \"66456109-7df3-4de7-978e-13496801fff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:00.885183 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.885132    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgx5z\" (UniqueName: \"kubernetes.io/projected/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-kube-api-access-jgx5z\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27\" (UID: \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:00.885255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.885179    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2wp\" (UniqueName: \"kubernetes.io/projected/66456109-7df3-4de7-978e-13496801fff2-kube-api-access-xn2wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h\" (UID: \"66456109-7df3-4de7-978e-13496801fff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:00.885407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.885269    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a541-f5aa-4272-bbe1-c75fd95e384a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8\" (UID: \"9325a541-f5aa-4272-bbe1-c75fd95e384a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:00.910739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.910635    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv696\" (UniqueName: \"kubernetes.io/projected/552e7cb2-282d-4f7e-b4ca-952b9df7a041-kube-api-access-wv696\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg\" (UID: \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:00.915281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.915246    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bc62a2-0765-497f-8529-8a11acfeb6be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb\" (UID: \"a7bc62a2-0765-497f-8529-8a11acfeb6be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:00.915421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.915325    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd449\" (UniqueName: \"kubernetes.io/projected/816d95a5-15ed-418c-93f1-79b39600a5e3-kube-api-access-jd449\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh\" (UID: \"816d95a5-15ed-418c-93f1-79b39600a5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:00.915421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.915397    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb\" (UID: \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:00.915543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.915461    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d95a5-15ed-418c-93f1-79b39600a5e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh\" (UID: \"816d95a5-15ed-418c-93f1-79b39600a5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:00.915599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.915578    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbw4\" (UniqueName: \"kubernetes.io/projected/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-kube-api-access-hjbw4\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb\" (UID: \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:00.915834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.887046    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:00.929451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.898788    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e7cb2-282d-4f7e-b4ca-952b9df7a041-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg\" (UID: \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:00.929451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.899485    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589327-9884-4386-a522-df5b12d23094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg\" (UID: \"49589327-9884-4386-a522-df5b12d23094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:00.929959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.895458    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:00.930080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.930042    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d95a5-15ed-418c-93f1-79b39600a5e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh\" (UID: \"816d95a5-15ed-418c-93f1-79b39600a5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:00.934060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.896430    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:00.934749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.897573    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27\" (UID: \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:00.941409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.941364    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcqlb\" (UniqueName: \"kubernetes.io/projected/49589327-9884-4386-a522-df5b12d23094-kube-api-access-gcqlb\") pod \"maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg\" (UID: \"49589327-9884-4386-a522-df5b12d23094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:00.941614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.941588    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66456109-7df3-4de7-978e-13496801fff2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h\" (UID: \"66456109-7df3-4de7-978e-13496801fff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:00.943308 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.943280    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2wp\" (UniqueName: \"kubernetes.io/projected/66456109-7df3-4de7-978e-13496801fff2-kube-api-access-xn2wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h\" (UID: \"66456109-7df3-4de7-978e-13496801fff2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:00.946330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.946239    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv696\" (UniqueName: \"kubernetes.io/projected/552e7cb2-282d-4f7e-b4ca-952b9df7a041-kube-api-access-wv696\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg\" (UID: \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:00.951675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.950728    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:00.980836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.980797    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"]
Oct 30 13:45:00.980990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.980867    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"]
Oct 30 13:45:00.980990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.980912    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"]
Oct 30 13:45:00.984388 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.984245    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:00.988955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.988924    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgx5z\" (UniqueName: \"kubernetes.io/projected/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-kube-api-access-jgx5z\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27\" (UID: \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:00.995309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:00.995250    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd449\" (UniqueName: \"kubernetes.io/projected/816d95a5-15ed-418c-93f1-79b39600a5e3-kube-api-access-jd449\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh\" (UID: \"816d95a5-15ed-418c-93f1-79b39600a5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:01.005127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.003384    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"]
Oct 30 13:45:01.005127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.003444    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"]
Oct 30 13:45:01.005127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.003469    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"]
Oct 30 13:45:01.005127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.003493    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"]
Oct 30 13:45:01.005127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.004060    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:01.020456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.020089    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021077    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a541-f5aa-4272-bbe1-c75fd95e384a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8\" (UID: \"9325a541-f5aa-4272-bbe1-c75fd95e384a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021223    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bc62a2-0765-497f-8529-8a11acfeb6be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb\" (UID: \"a7bc62a2-0765-497f-8529-8a11acfeb6be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021280    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb\" (UID: \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021352    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbw4\" (UniqueName: \"kubernetes.io/projected/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-kube-api-access-hjbw4\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb\" (UID: \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021394    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ac7f13-0f86-43e3-9189-d89aa39defb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp\" (UID: \"59ac7f13-0f86-43e3-9189-d89aa39defb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021533    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdk5\" (UniqueName: \"kubernetes.io/projected/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-kube-api-access-cgdk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh\" (UID: \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021583    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzjs\" (UniqueName: \"kubernetes.io/projected/59ac7f13-0f86-43e3-9189-d89aa39defb9-kube-api-access-qkzjs\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp\" (UID: \"59ac7f13-0f86-43e3-9189-d89aa39defb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021612    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh\" (UID: \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021673    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpv4f\" (UniqueName: \"kubernetes.io/projected/a7bc62a2-0765-497f-8529-8a11acfeb6be-kube-api-access-lpv4f\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb\" (UID: \"a7bc62a2-0765-497f-8529-8a11acfeb6be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.021732    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf2c6\" (UniqueName: \"kubernetes.io/projected/9325a541-f5aa-4272-bbe1-c75fd95e384a-kube-api-access-jf2c6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8\" (UID: \"9325a541-f5aa-4272-bbe1-c75fd95e384a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.022353    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb\" (UID: \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.023018    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a541-f5aa-4272-bbe1-c75fd95e384a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8\" (UID: \"9325a541-f5aa-4272-bbe1-c75fd95e384a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:01.023296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.023039    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ac7f13-0f86-43e3-9189-d89aa39defb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp\" (UID: \"59ac7f13-0f86-43e3-9189-d89aa39defb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:01.032660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.023762    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bc62a2-0765-497f-8529-8a11acfeb6be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb\" (UID: \"a7bc62a2-0765-497f-8529-8a11acfeb6be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:01.032660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.030314    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"]
Oct 30 13:45:01.032660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.030362    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"]
Oct 30 13:45:01.037530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.037481    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:01.051696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.051543    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:01.051883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.051863    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"]
Oct 30 13:45:01.052026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.052013    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"]
Oct 30 13:45:01.068818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.068749    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"]
Oct 30 13:45:01.093057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.091977    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9a93bd/test-model-9a93bd-kserve-55f9488bd7-6n79p"]
Oct 30 13:45:01.104801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.104767    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpv4f\" (UniqueName: \"kubernetes.io/projected/a7bc62a2-0765-497f-8529-8a11acfeb6be-kube-api-access-lpv4f\") pod \"maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb\" (UID: \"a7bc62a2-0765-497f-8529-8a11acfeb6be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:01.105547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.105508    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzjs\" (UniqueName: \"kubernetes.io/projected/59ac7f13-0f86-43e3-9189-d89aa39defb9-kube-api-access-qkzjs\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp\" (UID: \"59ac7f13-0f86-43e3-9189-d89aa39defb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:01.117319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.117289    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbw4\" (UniqueName: \"kubernetes.io/projected/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-kube-api-access-hjbw4\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb\" (UID: \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:01.118774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.118696    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf2c6\" (UniqueName: \"kubernetes.io/projected/9325a541-f5aa-4272-bbe1-c75fd95e384a-kube-api-access-jf2c6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8\" (UID: \"9325a541-f5aa-4272-bbe1-c75fd95e384a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:01.124063 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.123193    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zd82\" (UniqueName: \"kubernetes.io/projected/7febc0c4-d800-4a15-b9b8-708d4517a176-kube-api-access-6zd82\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz\" (UID: \"7febc0c4-d800-4a15-b9b8-708d4517a176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:01.124201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.124165    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49c1907-0584-44be-8022-3b27cc76ec35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2\" (UID: \"f49c1907-0584-44be-8022-3b27cc76ec35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:01.124365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.124247    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9wr\" (UniqueName: \"kubernetes.io/projected/d8394c36-ec95-44a0-a6d3-a01716310ebc-kube-api-access-tg9wr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2\" (UID: \"d8394c36-ec95-44a0-a6d3-a01716310ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:01.124365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.124337    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7febc0c4-d800-4a15-b9b8-708d4517a176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz\" (UID: \"7febc0c4-d800-4a15-b9b8-708d4517a176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:01.125238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.124541    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8394c36-ec95-44a0-a6d3-a01716310ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2\" (UID: \"d8394c36-ec95-44a0-a6d3-a01716310ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:01.125238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.124608    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdk5\" (UniqueName: \"kubernetes.io/projected/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-kube-api-access-cgdk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh\" (UID: \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:01.125238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.124653    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh\" (UID: \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:01.125238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.124678    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2w2t\" (UniqueName: \"kubernetes.io/projected/f49c1907-0584-44be-8022-3b27cc76ec35-kube-api-access-m2w2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2\" (UID: \"f49c1907-0584-44be-8022-3b27cc76ec35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:01.139655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.139594    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh\" (UID: \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:01.152746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.152709    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:01.169919 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.169658    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:01.185263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.177899    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdk5\" (UniqueName: \"kubernetes.io/projected/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-kube-api-access-cgdk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh\" (UID: \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:01.209474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.201789    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:01.209474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.206189    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:01.215484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.215453    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:01.225851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.225817    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8394c36-ec95-44a0-a6d3-a01716310ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2\" (UID: \"d8394c36-ec95-44a0-a6d3-a01716310ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:01.225945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.225925    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2w2t\" (UniqueName: \"kubernetes.io/projected/f49c1907-0584-44be-8022-3b27cc76ec35-kube-api-access-m2w2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2\" (UID: \"f49c1907-0584-44be-8022-3b27cc76ec35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:01.226795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.226523    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zd82\" (UniqueName: \"kubernetes.io/projected/7febc0c4-d800-4a15-b9b8-708d4517a176-kube-api-access-6zd82\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz\" (UID: \"7febc0c4-d800-4a15-b9b8-708d4517a176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:01.226924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.226815    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8394c36-ec95-44a0-a6d3-a01716310ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2\" (UID: \"d8394c36-ec95-44a0-a6d3-a01716310ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:01.227549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.227118    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49c1907-0584-44be-8022-3b27cc76ec35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2\" (UID: \"f49c1907-0584-44be-8022-3b27cc76ec35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:01.227549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.227279    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9wr\" (UniqueName: \"kubernetes.io/projected/d8394c36-ec95-44a0-a6d3-a01716310ebc-kube-api-access-tg9wr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2\" (UID: \"d8394c36-ec95-44a0-a6d3-a01716310ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:01.227549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.227371    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7febc0c4-d800-4a15-b9b8-708d4517a176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz\" (UID: \"7febc0c4-d800-4a15-b9b8-708d4517a176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:01.228564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.228540    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7febc0c4-d800-4a15-b9b8-708d4517a176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz\" (UID: \"7febc0c4-d800-4a15-b9b8-708d4517a176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:01.228702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.228677    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49c1907-0584-44be-8022-3b27cc76ec35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2\" (UID: \"f49c1907-0584-44be-8022-3b27cc76ec35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:01.267071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.261135    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9wr\" (UniqueName: \"kubernetes.io/projected/d8394c36-ec95-44a0-a6d3-a01716310ebc-kube-api-access-tg9wr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2\" (UID: \"d8394c36-ec95-44a0-a6d3-a01716310ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:01.267071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.261743    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zd82\" (UniqueName: \"kubernetes.io/projected/7febc0c4-d800-4a15-b9b8-708d4517a176-kube-api-access-6zd82\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz\" (UID: \"7febc0c4-d800-4a15-b9b8-708d4517a176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:01.267071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.263815    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:01.290404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.283871    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2w2t\" (UniqueName: \"kubernetes.io/projected/f49c1907-0584-44be-8022-3b27cc76ec35-kube-api-access-m2w2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2\" (UID: \"f49c1907-0584-44be-8022-3b27cc76ec35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:01.290404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.286014    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:01.322997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.322959    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:01.356677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.337905    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"]
Oct 30 13:45:01.356677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.354334    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"]
Oct 30 13:45:01.359412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.359327    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"]
Oct 30 13:45:01.368502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.368475    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:01.420169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.400855    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:01.423236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.422989    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:01.484754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.484674    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"]
Oct 30 13:45:01.545724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.545657    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" event={"ID":"7f0393d0-1fa2-4248-9d19-f31b08556afa","Type":"ContainerStarted","Data":"130b9a7dab2a8e72c179e325b783105ef7cb33182f07f373657863a1363d5c7e"}
Oct 30 13:45:01.578265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.578221    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4" event={"ID":"48a48c09-e2e1-4006-8d0c-2331997f7a85","Type":"ContainerStarted","Data":"c52ce7a84ec1e03677e0e165662c5a3c3a145a0bfd67a06c396284e5b4f6a929"}
Oct 30 13:45:01.617648 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.614860    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"]
Oct 30 13:45:01.637388 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.633310    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"]
Oct 30 13:45:01.640017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.639861    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"]
Oct 30 13:45:01.728066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.728029    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96f62f9-a6d6-459d-a3f3-59ea00bafaac" path="/var/lib/kubelet/pods/b96f62f9-a6d6-459d-a3f3-59ea00bafaac/volumes"
Oct 30 13:45:01.729289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:01.729262    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"]
Oct 30 13:45:01.793007 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:01.789467    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7f69c1_5545_4e9c_8aee_2046fb00d5c6.slice/crio-e868d47b6e323c37c2ff03b9c3cf30d2578452bc93755950da34d5218ee5dce4 WatchSource:0}: Error finding container e868d47b6e323c37c2ff03b9c3cf30d2578452bc93755950da34d5218ee5dce4: Status 404 returned error can't find the container with id e868d47b6e323c37c2ff03b9c3cf30d2578452bc93755950da34d5218ee5dce4
Oct 30 13:45:02.124763 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.124733    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"]
Oct 30 13:45:02.130543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.130512    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"]
Oct 30 13:45:02.209366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.209331    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"]
Oct 30 13:45:02.281947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.265643    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"]
Oct 30 13:45:02.305253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.305180    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"]
Oct 30 13:45:02.367882 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.367850    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"]
Oct 30 13:45:02.375675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.375634    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"]
Oct 30 13:45:02.390976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.390944    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"]
Oct 30 13:45:02.436026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.435967    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"]
Oct 30 13:45:02.439906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.439880    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"]
Oct 30 13:45:02.445746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.445716    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"]
Oct 30 13:45:02.450532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.450505    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"]
Oct 30 13:45:02.456190 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:02.456159    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998580a2_a1bf_4eb6_b8db_b3277af46957.slice/crio-0f8de1681ba801360fa150330262d97393105c0805c571173259328f2e80daad WatchSource:0}: Error finding container 0f8de1681ba801360fa150330262d97393105c0805c571173259328f2e80daad: Status 404 returned error can't find the container with id 0f8de1681ba801360fa150330262d97393105c0805c571173259328f2e80daad
Oct 30 13:45:02.467200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.467171    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"]
Oct 30 13:45:02.475603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.475575    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"]
Oct 30 13:45:02.480391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.480364    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"]
Oct 30 13:45:02.501221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.501180    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"]
Oct 30 13:45:02.650180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.650147    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" event={"ID":"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6","Type":"ContainerStarted","Data":"e868d47b6e323c37c2ff03b9c3cf30d2578452bc93755950da34d5218ee5dce4"}
Oct 30 13:45:02.652297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.652266    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" event={"ID":"ebf48e8e-316b-4250-9bae-9bcb736d3cb9","Type":"ContainerStarted","Data":"d6c90317f298002b288b63ca1232de7744460f0140464e0e2ab1ddf4a5edf431"}
Oct 30 13:45:02.655734 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.655704    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" event={"ID":"613f276e-cad4-40e0-84e8-b5c7e2524853","Type":"ContainerStarted","Data":"d01ea1512817a792e3201cff75475dc60ef86bdd680eb78966a81705b7d70113"}
Oct 30 13:45:02.657594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.657564    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27" event={"ID":"97c724a9-ac88-4e9e-9d50-c8bf38d366a8","Type":"ContainerStarted","Data":"37d2153c858f9328b2fa00c26aa57f1869cdd7deef6041cd4020d54d09348db0"}
Oct 30 13:45:02.660566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.660536    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" event={"ID":"2907709a-ad1d-4724-ab42-02ad7defb4b1","Type":"ContainerStarted","Data":"62bb626b3c735df708949bc56610124be044b8dbadd909d895964d52f73ffd1a"}
Oct 30 13:45:02.664147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.664118    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" event={"ID":"380b156b-fe53-460f-aa79-2a9f98e37bd2","Type":"ContainerStarted","Data":"2435018a9855d02592fea085e824a4e548d7e0809692c9ee7ac423492ddecbea"}
Oct 30 13:45:02.666290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.666033    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" event={"ID":"de1bfff9-9805-4b5a-a959-734a9a1570e8","Type":"ContainerStarted","Data":"7597a98fd8dea289a2b603d0c2698dc016d4438f09825da411457313e13eb8b2"}
Oct 30 13:45:02.669975 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:02.669881    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49589327_9884_4386_a522_df5b12d23094.slice/crio-1d5c711a8d355d3700e9fd5ac0c26a667356433ca6747eada70fb1ef395178c5 WatchSource:0}: Error finding container 1d5c711a8d355d3700e9fd5ac0c26a667356433ca6747eada70fb1ef395178c5: Status 404 returned error can't find the container with id 1d5c711a8d355d3700e9fd5ac0c26a667356433ca6747eada70fb1ef395178c5
Oct 30 13:45:02.671304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.671195    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" event={"ID":"b96952ed-4208-4568-9005-6a53e3f36eb6","Type":"ContainerStarted","Data":"70c90cb30fd2914ae7fb7194db0a2f0217ff9cdd5b83ad2e92d1fdbf1f31d818"}
Oct 30 13:45:02.675200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:02.675158    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" event={"ID":"998580a2-a1bf-4eb6-b8db-b3277af46957","Type":"ContainerStarted","Data":"0f8de1681ba801360fa150330262d97393105c0805c571173259328f2e80daad"}
Oct 30 13:45:02.769680 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:02.769098    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ac7f13_0f86_43e3_9189_d89aa39defb9.slice/crio-0d96afea2775b91be810259109a1f6b1c592375d00b8fac3f82582a4d4962c42 WatchSource:0}: Error finding container 0d96afea2775b91be810259109a1f6b1c592375d00b8fac3f82582a4d4962c42: Status 404 returned error can't find the container with id 0d96afea2775b91be810259109a1f6b1c592375d00b8fac3f82582a4d4962c42
Oct 30 13:45:02.776701 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:02.776664    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75087b72_ac15_4d81_bd96_659ac0f7b2ef.slice/crio-8dde1fbb68263fab505741949ae5c1f757888a422b72cdcafd5965f47834b7ab WatchSource:0}: Error finding container 8dde1fbb68263fab505741949ae5c1f757888a422b72cdcafd5965f47834b7ab: Status 404 returned error can't find the container with id 8dde1fbb68263fab505741949ae5c1f757888a422b72cdcafd5965f47834b7ab
Oct 30 13:45:02.785243 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:02.782741    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c00bd4_f3cd_413f_8e54_7e7c98db5ca0.slice/crio-d36b4b37a1533697dfdab91dfb95e86fcc3d619564217e7a8eb5c65184c0e216 WatchSource:0}: Error finding container d36b4b37a1533697dfdab91dfb95e86fcc3d619564217e7a8eb5c65184c0e216: Status 404 returned error can't find the container with id d36b4b37a1533697dfdab91dfb95e86fcc3d619564217e7a8eb5c65184c0e216
Oct 30 13:45:03.714599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.714560    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" event={"ID":"d8394c36-ec95-44a0-a6d3-a01716310ebc","Type":"ContainerStarted","Data":"018918d3dcac43b45790a9b103a58ae7cdfea1ff060bab7a18503e9a9063a45f"}
Oct 30 13:45:03.715016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.714626    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" event={"ID":"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0","Type":"ContainerStarted","Data":"767ff54c76619e282ff816bb9819bd85ce7c25af87945e1eea3241807a6f828f"}
Oct 30 13:45:03.715016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.714657    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" event={"ID":"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0","Type":"ContainerStarted","Data":"d36b4b37a1533697dfdab91dfb95e86fcc3d619564217e7a8eb5c65184c0e216"}
Oct 30 13:45:03.716315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.716291    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" event={"ID":"7febc0c4-d800-4a15-b9b8-708d4517a176","Type":"ContainerStarted","Data":"7bb0d8f363f73cba680c6199317c29a2583f83c7171eadcd1e19ac746f239a23"}
Oct 30 13:45:03.717958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.717934    4993 generic.go:356] "Generic (PLEG): container finished" podID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerID="e6f38d8011abbdbe72e2f0d0bf3970af43a9f98fd491f657098ea31d5e4be771" exitCode=60
Oct 30 13:45:03.718071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.718009    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" event={"ID":"7f0393d0-1fa2-4248-9d19-f31b08556afa","Type":"ContainerDied","Data":"e6f38d8011abbdbe72e2f0d0bf3970af43a9f98fd491f657098ea31d5e4be771"}
Oct 30 13:45:03.718285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.718269    4993 scope.go:122] "RemoveContainer" containerID="e6f38d8011abbdbe72e2f0d0bf3970af43a9f98fd491f657098ea31d5e4be771"
Oct 30 13:45:03.719875 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.719851    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" event={"ID":"2907709a-ad1d-4724-ab42-02ad7defb4b1","Type":"ContainerStarted","Data":"529b853b80f3056aafcd0502b205ac1de57e52ba37b9716dd8bf86bf92c74024"}
Oct 30 13:45:03.722283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.722263    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" event={"ID":"380b156b-fe53-460f-aa79-2a9f98e37bd2","Type":"ContainerStarted","Data":"63faa31eb09694a0c81e50c773f98aeca6f18726ba81b816f57c7f43d9bfd446"}
Oct 30 13:45:03.722533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.722516    4993 scope.go:122] "RemoveContainer" containerID="63faa31eb09694a0c81e50c773f98aeca6f18726ba81b816f57c7f43d9bfd446"
Oct 30 13:45:03.724381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.724324    4993 generic.go:356] "Generic (PLEG): container finished" podID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerID="ef87b066ea5a7dc85bfe4d4a05a2cea0a3937d9d0de8b6ddb1269241dd5c54aa" exitCode=60
Oct 30 13:45:03.724381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.724348    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" event={"ID":"de1bfff9-9805-4b5a-a959-734a9a1570e8","Type":"ContainerDied","Data":"ef87b066ea5a7dc85bfe4d4a05a2cea0a3937d9d0de8b6ddb1269241dd5c54aa"}
Oct 30 13:45:03.724694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.724633    4993 scope.go:122] "RemoveContainer" containerID="ef87b066ea5a7dc85bfe4d4a05a2cea0a3937d9d0de8b6ddb1269241dd5c54aa"
Oct 30 13:45:03.726939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.726738    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" event={"ID":"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2","Type":"ContainerStarted","Data":"60b34d1f556b00c9840e4aebb75b19dc72092864e2fc71df6902cf64134021b1"}
Oct 30 13:45:03.728134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.728104    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" event={"ID":"49589327-9884-4386-a522-df5b12d23094","Type":"ContainerStarted","Data":"1d5c711a8d355d3700e9fd5ac0c26a667356433ca6747eada70fb1ef395178c5"}
Oct 30 13:45:03.729496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.729473    4993 generic.go:356] "Generic (PLEG): container finished" podID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerID="1f799546fe3068d934db776d830a4384b1dbd562cb343ee1b01e75c1b7a25d9e" exitCode=60
Oct 30 13:45:03.729596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.729554    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" event={"ID":"b96952ed-4208-4568-9005-6a53e3f36eb6","Type":"ContainerDied","Data":"1f799546fe3068d934db776d830a4384b1dbd562cb343ee1b01e75c1b7a25d9e"}
Oct 30 13:45:03.729768 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.729751    4993 scope.go:122] "RemoveContainer" containerID="1f799546fe3068d934db776d830a4384b1dbd562cb343ee1b01e75c1b7a25d9e"
Oct 30 13:45:03.730759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.730740    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" event={"ID":"66456109-7df3-4de7-978e-13496801fff2","Type":"ContainerStarted","Data":"e602d22a4538dcec1132463914cecd277a5d539b566898197d754b6e93c8c877"}
Oct 30 13:45:03.732571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.732540    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" event={"ID":"998580a2-a1bf-4eb6-b8db-b3277af46957","Type":"ContainerStarted","Data":"da7dac8114c5baaa8f0a80e9cef9349c6e77638b9da0b9824cee8bec1c7a52a6"}
Oct 30 13:45:03.733888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.733863    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" event={"ID":"59ac7f13-0f86-43e3-9189-d89aa39defb9","Type":"ContainerStarted","Data":"0d96afea2775b91be810259109a1f6b1c592375d00b8fac3f82582a4d4962c42"}
Oct 30 13:45:03.736395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.736367    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" event={"ID":"f49c1907-0584-44be-8022-3b27cc76ec35","Type":"ContainerStarted","Data":"ffac1c9fd365afa2b181ef4246dbb84597e7b02b897a89d5faed63c5111fd7f9"}
Oct 30 13:45:03.739357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.739336    4993 generic.go:356] "Generic (PLEG): container finished" podID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerID="7381761527af16464cfc04d935fe7ff875ba15dcc01ed0fe3430024fecd120c5" exitCode=60
Oct 30 13:45:03.739528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.739412    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" event={"ID":"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6","Type":"ContainerDied","Data":"7381761527af16464cfc04d935fe7ff875ba15dcc01ed0fe3430024fecd120c5"}
Oct 30 13:45:03.739666 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.739646    4993 scope.go:122] "RemoveContainer" containerID="7381761527af16464cfc04d935fe7ff875ba15dcc01ed0fe3430024fecd120c5"
Oct 30 13:45:03.741322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.741302    4993 generic.go:356] "Generic (PLEG): container finished" podID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerID="649a526b330774c3cfe042b546f0863126946256e19d7af2c6004f9119c18717" exitCode=60
Oct 30 13:45:03.741493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.741474    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" event={"ID":"ebf48e8e-316b-4250-9bae-9bcb736d3cb9","Type":"ContainerDied","Data":"649a526b330774c3cfe042b546f0863126946256e19d7af2c6004f9119c18717"}
Oct 30 13:45:03.741911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.741627    4993 scope.go:122] "RemoveContainer" containerID="649a526b330774c3cfe042b546f0863126946256e19d7af2c6004f9119c18717"
Oct 30 13:45:03.743643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.743619    4993 generic.go:356] "Generic (PLEG): container finished" podID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerID="92bf01cf2a8e4236d6ce5ab0bd5000708d9c3279ff8e086dafe5774f59663297" exitCode=60
Oct 30 13:45:03.743741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.743702    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" event={"ID":"613f276e-cad4-40e0-84e8-b5c7e2524853","Type":"ContainerDied","Data":"92bf01cf2a8e4236d6ce5ab0bd5000708d9c3279ff8e086dafe5774f59663297"}
Oct 30 13:45:03.743920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.743903    4993 scope.go:122] "RemoveContainer" containerID="92bf01cf2a8e4236d6ce5ab0bd5000708d9c3279ff8e086dafe5774f59663297"
Oct 30 13:45:03.745256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.745230    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" event={"ID":"75087b72-ac15-4d81-bd96-659ac0f7b2ef","Type":"ContainerStarted","Data":"8dde1fbb68263fab505741949ae5c1f757888a422b72cdcafd5965f47834b7ab"}
Oct 30 13:45:03.747825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.747098    4993 generic.go:356] "Generic (PLEG): container finished" podID="48a48c09-e2e1-4006-8d0c-2331997f7a85" containerID="cc1873e73149fe45d6bbf3b23b84c9ff74927fbccd5f2190bd3922efd1c50d33" exitCode=60
Oct 30 13:45:03.747825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.747164    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4" event={"ID":"48a48c09-e2e1-4006-8d0c-2331997f7a85","Type":"ContainerDied","Data":"cc1873e73149fe45d6bbf3b23b84c9ff74927fbccd5f2190bd3922efd1c50d33"}
Oct 30 13:45:03.747825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.747494    4993 scope.go:122] "RemoveContainer" containerID="cc1873e73149fe45d6bbf3b23b84c9ff74927fbccd5f2190bd3922efd1c50d33"
Oct 30 13:45:03.748826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.748800    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" event={"ID":"a7bc62a2-0765-497f-8529-8a11acfeb6be","Type":"ContainerStarted","Data":"d1a5f6996523c9e80d74ba014df31780ef7b3a2ef811a7edf2c5975a14ced408"}
Oct 30 13:45:03.750257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.750233    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" event={"ID":"552e7cb2-282d-4f7e-b4ca-952b9df7a041","Type":"ContainerStarted","Data":"74e28f420a3d1254d51c5bc39a3c9960f933ad80e765f7def5ce6bf2c3258855"}
Oct 30 13:45:03.757327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.757274    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" podStartSLOduration=3.7572563150000002 podStartE2EDuration="3.757256315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.751853492 +0000 UTC m=+4368.227573104" watchObservedRunningTime="2026-10-30 13:45:03.757256315 +0000 UTC m=+4368.232975925"
Oct 30 13:45:03.773166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.773042    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" event={"ID":"816d95a5-15ed-418c-93f1-79b39600a5e3","Type":"ContainerStarted","Data":"8fe244a9bbfcd84fcdc7fe7405b0e89dc4ff993788e533512a2b4ff57049f3a8"}
Oct 30 13:45:03.773166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.773099    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" event={"ID":"816d95a5-15ed-418c-93f1-79b39600a5e3","Type":"ContainerStarted","Data":"ddb92439175bbffc41543c5c2646b4a30539c98c13e106c04fb5f72af7c93504"}
Oct 30 13:45:03.786228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.786124    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" event={"ID":"9325a541-f5aa-4272-bbe1-c75fd95e384a","Type":"ContainerStarted","Data":"a1de8fefcb78d2dc6e4a07f3596f4b46fbd655f8b38bc158ac46f487da7fd28f"}
Oct 30 13:45:03.835997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.835939    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" podStartSLOduration=3.835904722 podStartE2EDuration="3.835904722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.832589628 +0000 UTC m=+4368.308309240" watchObservedRunningTime="2026-10-30 13:45:03.835904722 +0000 UTC m=+4368.311624334"
Oct 30 13:45:03.916489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:03.916418    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" podStartSLOduration=3.91640475 podStartE2EDuration="3.91640475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.911264489 +0000 UTC m=+4368.386984097" watchObservedRunningTime="2026-10-30 13:45:03.91640475 +0000 UTC m=+4368.392124361"
Oct 30 13:45:04.172660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.172587    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" podStartSLOduration=4.172560572 podStartE2EDuration="4.172560572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.160771152 +0000 UTC m=+4368.636490763" watchObservedRunningTime="2026-10-30 13:45:04.172560572 +0000 UTC m=+4368.648280182"
Oct 30 13:45:04.259459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.256320    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27" podStartSLOduration=4.2562880100000005 podStartE2EDuration="4.25628801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.245348063 +0000 UTC m=+4368.721067673" watchObservedRunningTime="2026-10-30 13:45:04.25628801 +0000 UTC m=+4368.732007620"
Oct 30 13:45:04.351853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.350326    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" podStartSLOduration=4.350304707 podStartE2EDuration="4.350304707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.297544281 +0000 UTC m=+4368.773263896" watchObservedRunningTime="2026-10-30 13:45:04.350304707 +0000 UTC m=+4368.826024317"
Oct 30 13:45:04.800172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.800131    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" event={"ID":"d8394c36-ec95-44a0-a6d3-a01716310ebc","Type":"ContainerStarted","Data":"086d900c6e02d53d98cc09c35c0d0dd1671d97c28490c50141d3567a23c0cbda"}
Oct 30 13:45:04.802545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.802511    4993 generic.go:356] "Generic (PLEG): container finished" podID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerID="767ff54c76619e282ff816bb9819bd85ce7c25af87945e1eea3241807a6f828f" exitCode=60
Oct 30 13:45:04.802683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.802592    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" event={"ID":"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0","Type":"ContainerDied","Data":"767ff54c76619e282ff816bb9819bd85ce7c25af87945e1eea3241807a6f828f"}
Oct 30 13:45:04.802936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.802876    4993 scope.go:122] "RemoveContainer" containerID="767ff54c76619e282ff816bb9819bd85ce7c25af87945e1eea3241807a6f828f"
Oct 30 13:45:04.804532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.804508    4993 generic.go:356] "Generic (PLEG): container finished" podID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerID="1feff11508a7de9561e83ab2a7533423347af12d5f329a2be064b643a86c412d" exitCode=60
Oct 30 13:45:04.804653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.804575    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27" event={"ID":"97c724a9-ac88-4e9e-9d50-c8bf38d366a8","Type":"ContainerDied","Data":"1feff11508a7de9561e83ab2a7533423347af12d5f329a2be064b643a86c412d"}
Oct 30 13:45:04.804969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.804913    4993 scope.go:122] "RemoveContainer" containerID="1feff11508a7de9561e83ab2a7533423347af12d5f329a2be064b643a86c412d"
Oct 30 13:45:04.806499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.806461    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" event={"ID":"7febc0c4-d800-4a15-b9b8-708d4517a176","Type":"ContainerStarted","Data":"72fab3a96379f8d6010ba4560963eed10074cf550d634585130896a2360762b7"}
Oct 30 13:45:04.808247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.808193    4993 generic.go:356] "Generic (PLEG): container finished" podID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerID="529b853b80f3056aafcd0502b205ac1de57e52ba37b9716dd8bf86bf92c74024" exitCode=60
Oct 30 13:45:04.808344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.808290    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" event={"ID":"2907709a-ad1d-4724-ab42-02ad7defb4b1","Type":"ContainerDied","Data":"529b853b80f3056aafcd0502b205ac1de57e52ba37b9716dd8bf86bf92c74024"}
Oct 30 13:45:04.808742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.808621    4993 scope.go:122] "RemoveContainer" containerID="529b853b80f3056aafcd0502b205ac1de57e52ba37b9716dd8bf86bf92c74024"
Oct 30 13:45:04.810020 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.810000    4993 generic.go:356] "Generic (PLEG): container finished" podID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerID="63faa31eb09694a0c81e50c773f98aeca6f18726ba81b816f57c7f43d9bfd446" exitCode=60
Oct 30 13:45:04.810132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.810068    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" event={"ID":"380b156b-fe53-460f-aa79-2a9f98e37bd2","Type":"ContainerDied","Data":"63faa31eb09694a0c81e50c773f98aeca6f18726ba81b816f57c7f43d9bfd446"}
Oct 30 13:45:04.812097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.812071    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" event={"ID":"de1bfff9-9805-4b5a-a959-734a9a1570e8","Type":"ContainerStarted","Data":"1580b838891b13f3aef1d83e4b4112aa1424a6e9c58436d602c5813c861a7887"}
Oct 30 13:45:04.816780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.816633    4993 generic.go:356] "Generic (PLEG): container finished" podID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerID="d43c1c0f7395cd07e1f3031dc64f3efeed7e09c0f20467c0fb70311170c587eb" exitCode=60
Oct 30 13:45:04.816780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.816740    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" event={"ID":"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2","Type":"ContainerDied","Data":"d43c1c0f7395cd07e1f3031dc64f3efeed7e09c0f20467c0fb70311170c587eb"}
Oct 30 13:45:04.816968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.816948    4993 scope.go:122] "RemoveContainer" containerID="d43c1c0f7395cd07e1f3031dc64f3efeed7e09c0f20467c0fb70311170c587eb"
Oct 30 13:45:04.818974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.818951    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" event={"ID":"49589327-9884-4386-a522-df5b12d23094","Type":"ContainerStarted","Data":"9ad42ee23c53f97dc1c95bf24ecd9034b53c98e62df002708cec2459f904737e"}
Oct 30 13:45:04.821805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.821778    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" event={"ID":"66456109-7df3-4de7-978e-13496801fff2","Type":"ContainerStarted","Data":"9a810f0776e70ead9f8608abc0c67ec8cd983b7805db463e66713a5472a6527b"}
Oct 30 13:45:04.823493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.823473    4993 generic.go:356] "Generic (PLEG): container finished" podID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerID="da7dac8114c5baaa8f0a80e9cef9349c6e77638b9da0b9824cee8bec1c7a52a6" exitCode=60
Oct 30 13:45:04.823615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.823550    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" event={"ID":"998580a2-a1bf-4eb6-b8db-b3277af46957","Type":"ContainerDied","Data":"da7dac8114c5baaa8f0a80e9cef9349c6e77638b9da0b9824cee8bec1c7a52a6"}
Oct 30 13:45:04.823796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.823773    4993 scope.go:122] "RemoveContainer" containerID="da7dac8114c5baaa8f0a80e9cef9349c6e77638b9da0b9824cee8bec1c7a52a6"
Oct 30 13:45:04.827140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.826481    4993 generic.go:356] "Generic (PLEG): container finished" podID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerID="beb892469c048dc16ec75ac0dd19c4bdfdf20771331e284c08d55cd40366e688" exitCode=60
Oct 30 13:45:04.827140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.826560    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" event={"ID":"59ac7f13-0f86-43e3-9189-d89aa39defb9","Type":"ContainerDied","Data":"beb892469c048dc16ec75ac0dd19c4bdfdf20771331e284c08d55cd40366e688"}
Oct 30 13:45:04.827140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.826776    4993 scope.go:122] "RemoveContainer" containerID="beb892469c048dc16ec75ac0dd19c4bdfdf20771331e284c08d55cd40366e688"
Oct 30 13:45:04.837101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.837061    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" event={"ID":"f49c1907-0584-44be-8022-3b27cc76ec35","Type":"ContainerStarted","Data":"1f0df0c136ece1c917d124d242f1cdb482859d3f9912b59f68f148833bb8d4b3"}
Oct 30 13:45:04.841747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.841724    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" event={"ID":"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6","Type":"ContainerStarted","Data":"8791be731685325c996fb68ef2c606aa1119745ab3c5ef70bb7422d21bbfc57b"}
Oct 30 13:45:04.843431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.843397    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" event={"ID":"75087b72-ac15-4d81-bd96-659ac0f7b2ef","Type":"ContainerStarted","Data":"c3a03f572d55378cdff05fad335136708d876e26f6a30485a857917c80257077"}
Oct 30 13:45:04.845161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.845135    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" event={"ID":"a7bc62a2-0765-497f-8529-8a11acfeb6be","Type":"ContainerStarted","Data":"e49d928eb247a3398af6c9269bf618c86b308b216c2f4185278e95fbd05774fa"}
Oct 30 13:45:04.846988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.846961    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" event={"ID":"552e7cb2-282d-4f7e-b4ca-952b9df7a041","Type":"ContainerStarted","Data":"7f5a1b3af0c71acd1b09395bfff5d01b4822c0bfb07566dc171bf79c4a8564e8"}
Oct 30 13:45:04.848890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.848863    4993 generic.go:356] "Generic (PLEG): container finished" podID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerID="8fe244a9bbfcd84fcdc7fe7405b0e89dc4ff993788e533512a2b4ff57049f3a8" exitCode=60
Oct 30 13:45:04.849028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.848942    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" event={"ID":"816d95a5-15ed-418c-93f1-79b39600a5e3","Type":"ContainerDied","Data":"8fe244a9bbfcd84fcdc7fe7405b0e89dc4ff993788e533512a2b4ff57049f3a8"}
Oct 30 13:45:04.849394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.849370    4993 scope.go:122] "RemoveContainer" containerID="8fe244a9bbfcd84fcdc7fe7405b0e89dc4ff993788e533512a2b4ff57049f3a8"
Oct 30 13:45:04.850141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.850105    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" podStartSLOduration=4.850094287 podStartE2EDuration="4.850094287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.848164761 +0000 UTC m=+4369.323884371" watchObservedRunningTime="2026-10-30 13:45:04.850094287 +0000 UTC m=+4369.325813897"
Oct 30 13:45:04.855562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.854945    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" event={"ID":"9325a541-f5aa-4272-bbe1-c75fd95e384a","Type":"ContainerStarted","Data":"3b09b6577f26d6a4b54ef67e904f2d647db53e3f30acdbb7f8679913faeebf4d"}
Oct 30 13:45:04.896077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.895116    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" podStartSLOduration=4.895097182 podStartE2EDuration="4.895097182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.889597499 +0000 UTC m=+4369.365317109" watchObservedRunningTime="2026-10-30 13:45:04.895097182 +0000 UTC m=+4369.370816794"
Oct 30 13:45:04.993373 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:04.993292    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" podStartSLOduration=4.993268995 podStartE2EDuration="4.993268995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.978734763 +0000 UTC m=+4369.454454385" watchObservedRunningTime="2026-10-30 13:45:04.993268995 +0000 UTC m=+4369.468988605"
Oct 30 13:45:05.111831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.111775    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" podStartSLOduration=5.111760768 podStartE2EDuration="5.111760768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.1079573 +0000 UTC m=+4369.583676910" watchObservedRunningTime="2026-10-30 13:45:05.111760768 +0000 UTC m=+4369.587480377"
Oct 30 13:45:05.198192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.198127    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" podStartSLOduration=5.198113338 podStartE2EDuration="5.198113338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.191594023 +0000 UTC m=+4369.667313633" watchObservedRunningTime="2026-10-30 13:45:05.198113338 +0000 UTC m=+4369.673832995"
Oct 30 13:45:05.239136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.239077    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" podStartSLOduration=5.23906184 podStartE2EDuration="5.23906184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.23601471 +0000 UTC m=+4369.711734317" watchObservedRunningTime="2026-10-30 13:45:05.23906184 +0000 UTC m=+4369.714781449"
Oct 30 13:45:05.286452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.286401    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" podStartSLOduration=5.286385262 podStartE2EDuration="5.286385262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.277754773 +0000 UTC m=+4369.753474387" watchObservedRunningTime="2026-10-30 13:45:05.286385262 +0000 UTC m=+4369.762104873"
Oct 30 13:45:05.332051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.331981    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" podStartSLOduration=5.331961208 podStartE2EDuration="5.331961208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.327351116 +0000 UTC m=+4369.803070727" watchObservedRunningTime="2026-10-30 13:45:05.331961208 +0000 UTC m=+4369.807680824"
Oct 30 13:45:05.418522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.418472    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" podStartSLOduration=5.418458555 podStartE2EDuration="5.418458555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.371958628 +0000 UTC m=+4369.847678238" watchObservedRunningTime="2026-10-30 13:45:05.418458555 +0000 UTC m=+4369.894178165"
Oct 30 13:45:05.421542 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.421505    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" podStartSLOduration=5.421493205 podStartE2EDuration="5.421493205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.413202756 +0000 UTC m=+4369.888986676" watchObservedRunningTime="2026-10-30 13:45:05.421493205 +0000 UTC m=+4369.897212815"
Oct 30 13:45:05.587708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.587646    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" podStartSLOduration=5.587629609 podStartE2EDuration="5.587629609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.581493116 +0000 UTC m=+4370.057212730" watchObservedRunningTime="2026-10-30 13:45:05.587629609 +0000 UTC m=+4370.063349219"
Oct 30 13:45:05.886187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.886149    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" event={"ID":"7f0393d0-1fa2-4248-9d19-f31b08556afa","Type":"ContainerStarted","Data":"d86bab65fba7dd522607859003bced0e3894cc12c424b887a1ff2d416759990e"}
Oct 30 13:45:05.889756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.889718    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" event={"ID":"2907709a-ad1d-4724-ab42-02ad7defb4b1","Type":"ContainerStarted","Data":"1748aaf25bb7978775531c0b742434109882d2457e718cfb933791a52102d0a9"}
Oct 30 13:45:05.892267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.892235    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" event={"ID":"380b156b-fe53-460f-aa79-2a9f98e37bd2","Type":"ContainerStarted","Data":"18255b0eb7c8cb0653a1e79fbf321f055d2cfdfd79a904b9c3a62b76213a5035"}
Oct 30 13:45:05.892615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.892597    4993 scope.go:122] "RemoveContainer" containerID="18255b0eb7c8cb0653a1e79fbf321f055d2cfdfd79a904b9c3a62b76213a5035"
Oct 30 13:45:05.893008 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:05.892985    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm_odh-ai-gateway-infra(380b156b-fe53-460f-aa79-2a9f98e37bd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2"
Oct 30 13:45:05.894676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.894649    4993 generic.go:356] "Generic (PLEG): container finished" podID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerID="1580b838891b13f3aef1d83e4b4112aa1424a6e9c58436d602c5813c861a7887" exitCode=60
Oct 30 13:45:05.894823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.894802    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" event={"ID":"de1bfff9-9805-4b5a-a959-734a9a1570e8","Type":"ContainerDied","Data":"1580b838891b13f3aef1d83e4b4112aa1424a6e9c58436d602c5813c861a7887"}
Oct 30 13:45:05.894893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.894857    4993 scope.go:122] "RemoveContainer" containerID="ef87b066ea5a7dc85bfe4d4a05a2cea0a3937d9d0de8b6ddb1269241dd5c54aa"
Oct 30 13:45:05.895371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.895021    4993 scope.go:122] "RemoveContainer" containerID="1580b838891b13f3aef1d83e4b4112aa1424a6e9c58436d602c5813c861a7887"
Oct 30 13:45:05.895668 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:05.895514    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm_odh-ai-gateway-infra(de1bfff9-9805-4b5a-a959-734a9a1570e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8"
Oct 30 13:45:05.897477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.897457    4993 generic.go:356] "Generic (PLEG): container finished" podID="49589327-9884-4386-a522-df5b12d23094" containerID="9ad42ee23c53f97dc1c95bf24ecd9034b53c98e62df002708cec2459f904737e" exitCode=60
Oct 30 13:45:05.897588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.897531    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" event={"ID":"49589327-9884-4386-a522-df5b12d23094","Type":"ContainerDied","Data":"9ad42ee23c53f97dc1c95bf24ecd9034b53c98e62df002708cec2459f904737e"}
Oct 30 13:45:05.897879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.897859    4993 scope.go:122] "RemoveContainer" containerID="9ad42ee23c53f97dc1c95bf24ecd9034b53c98e62df002708cec2459f904737e"
Oct 30 13:45:05.900368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.900330    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" event={"ID":"b96952ed-4208-4568-9005-6a53e3f36eb6","Type":"ContainerStarted","Data":"dce0d8a607e4a4df82f79c1964ef9d7575f82a432a17f4edc17d10821026efc1"}
Oct 30 13:45:05.902282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.902258    4993 generic.go:356] "Generic (PLEG): container finished" podID="66456109-7df3-4de7-978e-13496801fff2" containerID="9a810f0776e70ead9f8608abc0c67ec8cd983b7805db463e66713a5472a6527b" exitCode=60
Oct 30 13:45:05.902377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.902336    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" event={"ID":"66456109-7df3-4de7-978e-13496801fff2","Type":"ContainerDied","Data":"9a810f0776e70ead9f8608abc0c67ec8cd983b7805db463e66713a5472a6527b"}
Oct 30 13:45:05.902580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.902562    4993 scope.go:122] "RemoveContainer" containerID="9a810f0776e70ead9f8608abc0c67ec8cd983b7805db463e66713a5472a6527b"
Oct 30 13:45:05.904645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.904624    4993 generic.go:356] "Generic (PLEG): container finished" podID="f49c1907-0584-44be-8022-3b27cc76ec35" containerID="1f0df0c136ece1c917d124d242f1cdb482859d3f9912b59f68f148833bb8d4b3" exitCode=60
Oct 30 13:45:05.904735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.904709    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" event={"ID":"f49c1907-0584-44be-8022-3b27cc76ec35","Type":"ContainerDied","Data":"1f0df0c136ece1c917d124d242f1cdb482859d3f9912b59f68f148833bb8d4b3"}
Oct 30 13:45:05.904939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.904924    4993 scope.go:122] "RemoveContainer" containerID="1f0df0c136ece1c917d124d242f1cdb482859d3f9912b59f68f148833bb8d4b3"
Oct 30 13:45:05.906644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.906621    4993 generic.go:356] "Generic (PLEG): container finished" podID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerID="8791be731685325c996fb68ef2c606aa1119745ab3c5ef70bb7422d21bbfc57b" exitCode=60
Oct 30 13:45:05.906734 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.906702    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" event={"ID":"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6","Type":"ContainerDied","Data":"8791be731685325c996fb68ef2c606aa1119745ab3c5ef70bb7422d21bbfc57b"}
Oct 30 13:45:05.906935 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.906921    4993 scope.go:122] "RemoveContainer" containerID="8791be731685325c996fb68ef2c606aa1119745ab3c5ef70bb7422d21bbfc57b"
Oct 30 13:45:05.907255 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:05.907228    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782_odh-ai-gateway-infra(8e7f69c1-5545-4e9c-8aee-2046fb00d5c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6"
Oct 30 13:45:05.908703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.908681    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" event={"ID":"ebf48e8e-316b-4250-9bae-9bcb736d3cb9","Type":"ContainerStarted","Data":"94e714e37b94a4d1afd7e20af0efe6369d099001e46b1b395764e9e1c6b590b2"}
Oct 30 13:45:05.911229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.911195    4993 scope.go:122] "RemoveContainer" containerID="b46b3bc85e0e5e334033f07d21e3214049ea617f9010d62d3642fd53fc2f7a95"
Oct 30 13:45:05.911334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.911258    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" event={"ID":"613f276e-cad4-40e0-84e8-b5c7e2524853","Type":"ContainerStarted","Data":"b46b3bc85e0e5e334033f07d21e3214049ea617f9010d62d3642fd53fc2f7a95"}
Oct 30 13:45:05.911563 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:05.911543    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj_odh-ai-gateway-infra(613f276e-cad4-40e0-84e8-b5c7e2524853)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853"
Oct 30 13:45:05.913158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.913134    4993 generic.go:356] "Generic (PLEG): container finished" podID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerID="c3a03f572d55378cdff05fad335136708d876e26f6a30485a857917c80257077" exitCode=60
Oct 30 13:45:05.913315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.913274    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" event={"ID":"75087b72-ac15-4d81-bd96-659ac0f7b2ef","Type":"ContainerDied","Data":"c3a03f572d55378cdff05fad335136708d876e26f6a30485a857917c80257077"}
Oct 30 13:45:05.913806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.913770    4993 scope.go:122] "RemoveContainer" containerID="c3a03f572d55378cdff05fad335136708d876e26f6a30485a857917c80257077"
Oct 30 13:45:05.915642 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.915620    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4" event={"ID":"48a48c09-e2e1-4006-8d0c-2331997f7a85","Type":"ContainerStarted","Data":"6e09601149486d8cdfad5aff658d9fed266fef0319d4684bb47f23e7c23ddb0e"}
Oct 30 13:45:05.917103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.917081    4993 generic.go:356] "Generic (PLEG): container finished" podID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerID="e49d928eb247a3398af6c9269bf618c86b308b216c2f4185278e95fbd05774fa" exitCode=60
Oct 30 13:45:05.917179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.917152    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" event={"ID":"a7bc62a2-0765-497f-8529-8a11acfeb6be","Type":"ContainerDied","Data":"e49d928eb247a3398af6c9269bf618c86b308b216c2f4185278e95fbd05774fa"}
Oct 30 13:45:05.917352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.917337    4993 scope.go:122] "RemoveContainer" containerID="e49d928eb247a3398af6c9269bf618c86b308b216c2f4185278e95fbd05774fa"
Oct 30 13:45:05.918775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.918753    4993 generic.go:356] "Generic (PLEG): container finished" podID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerID="7f5a1b3af0c71acd1b09395bfff5d01b4822c0bfb07566dc171bf79c4a8564e8" exitCode=60
Oct 30 13:45:05.918918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.918779    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" event={"ID":"552e7cb2-282d-4f7e-b4ca-952b9df7a041","Type":"ContainerDied","Data":"7f5a1b3af0c71acd1b09395bfff5d01b4822c0bfb07566dc171bf79c4a8564e8"}
Oct 30 13:45:05.919026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.919011    4993 scope.go:122] "RemoveContainer" containerID="7f5a1b3af0c71acd1b09395bfff5d01b4822c0bfb07566dc171bf79c4a8564e8"
Oct 30 13:45:05.920346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.920323    4993 generic.go:356] "Generic (PLEG): container finished" podID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerID="3b09b6577f26d6a4b54ef67e904f2d647db53e3f30acdbb7f8679913faeebf4d" exitCode=60
Oct 30 13:45:05.920453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.920346    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" event={"ID":"9325a541-f5aa-4272-bbe1-c75fd95e384a","Type":"ContainerDied","Data":"3b09b6577f26d6a4b54ef67e904f2d647db53e3f30acdbb7f8679913faeebf4d"}
Oct 30 13:45:05.920615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.920598    4993 scope.go:122] "RemoveContainer" containerID="3b09b6577f26d6a4b54ef67e904f2d647db53e3f30acdbb7f8679913faeebf4d"
Oct 30 13:45:05.921990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.921970    4993 generic.go:356] "Generic (PLEG): container finished" podID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerID="086d900c6e02d53d98cc09c35c0d0dd1671d97c28490c50141d3567a23c0cbda" exitCode=60
Oct 30 13:45:05.922086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.921994    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" event={"ID":"d8394c36-ec95-44a0-a6d3-a01716310ebc","Type":"ContainerDied","Data":"086d900c6e02d53d98cc09c35c0d0dd1671d97c28490c50141d3567a23c0cbda"}
Oct 30 13:45:05.922285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.922267    4993 scope.go:122] "RemoveContainer" containerID="086d900c6e02d53d98cc09c35c0d0dd1671d97c28490c50141d3567a23c0cbda"
Oct 30 13:45:05.923612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.923595    4993 generic.go:356] "Generic (PLEG): container finished" podID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerID="72fab3a96379f8d6010ba4560963eed10074cf550d634585130896a2360762b7" exitCode=60
Oct 30 13:45:05.923694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.923625    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" event={"ID":"7febc0c4-d800-4a15-b9b8-708d4517a176","Type":"ContainerDied","Data":"72fab3a96379f8d6010ba4560963eed10074cf550d634585130896a2360762b7"}
Oct 30 13:45:05.923831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.923817    4993 scope.go:122] "RemoveContainer" containerID="72fab3a96379f8d6010ba4560963eed10074cf550d634585130896a2360762b7"
Oct 30 13:45:05.934663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.934622    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" podStartSLOduration=5.934610397 podStartE2EDuration="5.934610397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.929421609 +0000 UTC m=+4370.405141218" watchObservedRunningTime="2026-10-30 13:45:05.934610397 +0000 UTC m=+4370.410330006"
Oct 30 13:45:05.999449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:05.999352    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" podStartSLOduration=5.999330993 podStartE2EDuration="5.999330993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.995424051 +0000 UTC m=+4370.471143664" watchObservedRunningTime="2026-10-30 13:45:05.999330993 +0000 UTC m=+4370.475050603"
Oct 30 13:45:06.198917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.198781    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" podStartSLOduration=6.198743479 podStartE2EDuration="6.198743479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:06.157835557 +0000 UTC m=+4370.633555170" watchObservedRunningTime="2026-10-30 13:45:06.198743479 +0000 UTC m=+4370.674479824"
Oct 30 13:45:06.221574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.221249    4993 scope.go:122] "RemoveContainer" containerID="7381761527af16464cfc04d935fe7ff875ba15dcc01ed0fe3430024fecd120c5"
Oct 30 13:45:06.506324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.502320    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4" podStartSLOduration=6.502301929 podStartE2EDuration="6.502301929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:06.50083074 +0000 UTC m=+4370.976550351" watchObservedRunningTime="2026-10-30 13:45:06.502301929 +0000 UTC m=+4370.978021540"
Oct 30 13:45:06.936479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.936308    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"]
Oct 30 13:45:06.955298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.954437    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" event={"ID":"49589327-9884-4386-a522-df5b12d23094","Type":"ContainerStarted","Data":"093d43232dde3539ecb5a71e0daa7382f8ca1db68788b212819795224c73c74f"}
Oct 30 13:45:06.959101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.958077    4993 generic.go:356] "Generic (PLEG): container finished" podID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerID="dce0d8a607e4a4df82f79c1964ef9d7575f82a432a17f4edc17d10821026efc1" exitCode=60
Oct 30 13:45:06.959101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.958151    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" event={"ID":"b96952ed-4208-4568-9005-6a53e3f36eb6","Type":"ContainerDied","Data":"dce0d8a607e4a4df82f79c1964ef9d7575f82a432a17f4edc17d10821026efc1"}
Oct 30 13:45:06.959101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.958195    4993 scope.go:122] "RemoveContainer" containerID="1f799546fe3068d934db776d830a4384b1dbd562cb343ee1b01e75c1b7a25d9e"
Oct 30 13:45:06.959101 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:06.958673    4993 scope.go:122] "RemoveContainer" containerID="dce0d8a607e4a4df82f79c1964ef9d7575f82a432a17f4edc17d10821026efc1"
Oct 30 13:45:06.959101 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:06.959042    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8_odh-ai-gateway-infra(b96952ed-4208-4568-9005-6a53e3f36eb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6"
Oct 30 13:45:07.287549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.287432    4993 generic.go:356] "Generic (PLEG): container finished" podID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerID="835117598a783b80b0f477b7316de8825b438e2e5117cd1487b3f585bcc4a3e9" exitCode=60
Oct 30 13:45:07.287960 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.287929    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" event={"ID":"998580a2-a1bf-4eb6-b8db-b3277af46957","Type":"ContainerDied","Data":"835117598a783b80b0f477b7316de8825b438e2e5117cd1487b3f585bcc4a3e9"}
Oct 30 13:45:07.288823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.288802    4993 scope.go:122] "RemoveContainer" containerID="835117598a783b80b0f477b7316de8825b438e2e5117cd1487b3f585bcc4a3e9"
Oct 30 13:45:07.297493 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.297457    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9_odh-ai-gateway-infra(998580a2-a1bf-4eb6-b8db-b3277af46957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957"
Oct 30 13:45:07.359250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.353621    4993 generic.go:356] "Generic (PLEG): container finished" podID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerID="3a59798e0578c8bc91ef567f653b0d1201a3027734377aacf80ca36ebe2e4806" exitCode=60
Oct 30 13:45:07.364161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.362002    4993 scope.go:122] "RemoveContainer" containerID="3a59798e0578c8bc91ef567f653b0d1201a3027734377aacf80ca36ebe2e4806"
Oct 30 13:45:07.365159 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.365126    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp_odh-ai-gateway-infra(59ac7f13-0f86-43e3-9189-d89aa39defb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9"
Oct 30 13:45:07.365542 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.362380    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" event={"ID":"59ac7f13-0f86-43e3-9189-d89aa39defb9","Type":"ContainerDied","Data":"3a59798e0578c8bc91ef567f653b0d1201a3027734377aacf80ca36ebe2e4806"}
Oct 30 13:45:07.374878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.374693    4993 scope.go:122] "RemoveContainer" containerID="8791be731685325c996fb68ef2c606aa1119745ab3c5ef70bb7422d21bbfc57b"
Oct 30 13:45:07.376184 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.376152    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782_odh-ai-gateway-infra(8e7f69c1-5545-4e9c-8aee-2046fb00d5c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6"
Oct 30 13:45:07.380949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.380923    4993 generic.go:356] "Generic (PLEG): container finished" podID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerID="94e714e37b94a4d1afd7e20af0efe6369d099001e46b1b395764e9e1c6b590b2" exitCode=60
Oct 30 13:45:07.381218 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.381182    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" event={"ID":"ebf48e8e-316b-4250-9bae-9bcb736d3cb9","Type":"ContainerDied","Data":"94e714e37b94a4d1afd7e20af0efe6369d099001e46b1b395764e9e1c6b590b2"}
Oct 30 13:45:07.384684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.384660    4993 scope.go:122] "RemoveContainer" containerID="94e714e37b94a4d1afd7e20af0efe6369d099001e46b1b395764e9e1c6b590b2"
Oct 30 13:45:07.387138 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.387114    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw_odh-ai-gateway-infra(ebf48e8e-316b-4250-9bae-9bcb736d3cb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9"
Oct 30 13:45:07.387471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.387305    4993 generic.go:356] "Generic (PLEG): container finished" podID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerID="b46b3bc85e0e5e334033f07d21e3214049ea617f9010d62d3642fd53fc2f7a95" exitCode=60
Oct 30 13:45:07.387782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.387327    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" event={"ID":"613f276e-cad4-40e0-84e8-b5c7e2524853","Type":"ContainerDied","Data":"b46b3bc85e0e5e334033f07d21e3214049ea617f9010d62d3642fd53fc2f7a95"}
Oct 30 13:45:07.388120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.387373    4993 scope.go:122] "RemoveContainer" containerID="da7dac8114c5baaa8f0a80e9cef9349c6e77638b9da0b9824cee8bec1c7a52a6"
Oct 30 13:45:07.389108 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.389091    4993 scope.go:122] "RemoveContainer" containerID="b46b3bc85e0e5e334033f07d21e3214049ea617f9010d62d3642fd53fc2f7a95"
Oct 30 13:45:07.397809 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.397784    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj_odh-ai-gateway-infra(613f276e-cad4-40e0-84e8-b5c7e2524853)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853"
Oct 30 13:45:07.408014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.407049    4993 generic.go:356] "Generic (PLEG): container finished" podID="48a48c09-e2e1-4006-8d0c-2331997f7a85" containerID="6e09601149486d8cdfad5aff658d9fed266fef0319d4684bb47f23e7c23ddb0e" exitCode=60
Oct 30 13:45:07.408014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.407122    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4" event={"ID":"48a48c09-e2e1-4006-8d0c-2331997f7a85","Type":"ContainerDied","Data":"6e09601149486d8cdfad5aff658d9fed266fef0319d4684bb47f23e7c23ddb0e"}
Oct 30 13:45:07.408014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.407588    4993 scope.go:122] "RemoveContainer" containerID="6e09601149486d8cdfad5aff658d9fed266fef0319d4684bb47f23e7c23ddb0e"
Oct 30 13:45:07.408014 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.407960    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4_odh-ai-gateway-infra(48a48c09-e2e1-4006-8d0c-2331997f7a85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85"
Oct 30 13:45:07.412981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.412957    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" event={"ID":"a7bc62a2-0765-497f-8529-8a11acfeb6be","Type":"ContainerStarted","Data":"a102c923dfa34b61c5fe96732c04b51e6c4ea2762f0f731f76fe5c8657ec870e"}
Oct 30 13:45:07.419283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.418491    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" event={"ID":"552e7cb2-282d-4f7e-b4ca-952b9df7a041","Type":"ContainerStarted","Data":"a607b50c5d6427b9b38c6f660674d9cdf682513f46cfef94440825753ecd7cfe"}
Oct 30 13:45:07.421415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.421389    4993 generic.go:356] "Generic (PLEG): container finished" podID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerID="a97a40cd0d53f263564d082e6216291d65151270af5ea8640114cc2ab928c8e5" exitCode=60
Oct 30 13:45:07.421694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.421565    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" event={"ID":"816d95a5-15ed-418c-93f1-79b39600a5e3","Type":"ContainerDied","Data":"a97a40cd0d53f263564d082e6216291d65151270af5ea8640114cc2ab928c8e5"}
Oct 30 13:45:07.421792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.421697    4993 scope.go:122] "RemoveContainer" containerID="a97a40cd0d53f263564d082e6216291d65151270af5ea8640114cc2ab928c8e5"
Oct 30 13:45:07.422339 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.422309    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh_odh-ai-gateway-infra(816d95a5-15ed-418c-93f1-79b39600a5e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3"
Oct 30 13:45:07.425571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.424757    4993 generic.go:356] "Generic (PLEG): container finished" podID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerID="5b817eba7ecac6613ced24bcb35dd54101c82d26bacfde1a5e69ebb419612331" exitCode=60
Oct 30 13:45:07.425571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.424818    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" event={"ID":"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0","Type":"ContainerDied","Data":"5b817eba7ecac6613ced24bcb35dd54101c82d26bacfde1a5e69ebb419612331"}
Oct 30 13:45:07.425571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.425116    4993 scope.go:122] "RemoveContainer" containerID="5b817eba7ecac6613ced24bcb35dd54101c82d26bacfde1a5e69ebb419612331"
Oct 30 13:45:07.425571 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.425391    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb_odh-ai-gateway-infra(a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0"
Oct 30 13:45:07.427655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.427629    4993 generic.go:356] "Generic (PLEG): container finished" podID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerID="3345d718bb7bd8aec7d0a9ec5be1d09d98e5130e9559fd900342539c04505a24" exitCode=60
Oct 30 13:45:07.427761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.427726    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27" event={"ID":"97c724a9-ac88-4e9e-9d50-c8bf38d366a8","Type":"ContainerDied","Data":"3345d718bb7bd8aec7d0a9ec5be1d09d98e5130e9559fd900342539c04505a24"}
Oct 30 13:45:07.428127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.428106    4993 scope.go:122] "RemoveContainer" containerID="3345d718bb7bd8aec7d0a9ec5be1d09d98e5130e9559fd900342539c04505a24"
Oct 30 13:45:07.428611 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.428587    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27_odh-ai-gateway-infra(97c724a9-ac88-4e9e-9d50-c8bf38d366a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8"
Oct 30 13:45:07.432546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.432532    4993 scope.go:122] "RemoveContainer" containerID="beb892469c048dc16ec75ac0dd19c4bdfdf20771331e284c08d55cd40366e688"
Oct 30 13:45:07.436723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.436695    4993 generic.go:356] "Generic (PLEG): container finished" podID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerID="d86bab65fba7dd522607859003bced0e3894cc12c424b887a1ff2d416759990e" exitCode=60
Oct 30 13:45:07.436816 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.436768    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" event={"ID":"7f0393d0-1fa2-4248-9d19-f31b08556afa","Type":"ContainerDied","Data":"d86bab65fba7dd522607859003bced0e3894cc12c424b887a1ff2d416759990e"}
Oct 30 13:45:07.437011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.436991    4993 scope.go:122] "RemoveContainer" containerID="d86bab65fba7dd522607859003bced0e3894cc12c424b887a1ff2d416759990e"
Oct 30 13:45:07.437366 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.437343    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8_odh-ai-gateway-infra(7f0393d0-1fa2-4248-9d19-f31b08556afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa"
Oct 30 13:45:07.439150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.439125    4993 generic.go:356] "Generic (PLEG): container finished" podID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerID="1748aaf25bb7978775531c0b742434109882d2457e718cfb933791a52102d0a9" exitCode=60
Oct 30 13:45:07.439267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.439189    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" event={"ID":"2907709a-ad1d-4724-ab42-02ad7defb4b1","Type":"ContainerDied","Data":"1748aaf25bb7978775531c0b742434109882d2457e718cfb933791a52102d0a9"}
Oct 30 13:45:07.439468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.439450    4993 scope.go:122] "RemoveContainer" containerID="1748aaf25bb7978775531c0b742434109882d2457e718cfb933791a52102d0a9"
Oct 30 13:45:07.439739 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.439714    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8_odh-ai-gateway-infra(2907709a-ad1d-4724-ab42-02ad7defb4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1"
Oct 30 13:45:07.441159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.441135    4993 generic.go:356] "Generic (PLEG): container finished" podID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerID="18255b0eb7c8cb0653a1e79fbf321f055d2cfdfd79a904b9c3a62b76213a5035" exitCode=60
Oct 30 13:45:07.441270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.441224    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" event={"ID":"380b156b-fe53-460f-aa79-2a9f98e37bd2","Type":"ContainerDied","Data":"18255b0eb7c8cb0653a1e79fbf321f055d2cfdfd79a904b9c3a62b76213a5035"}
Oct 30 13:45:07.441548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.441525    4993 scope.go:122] "RemoveContainer" containerID="18255b0eb7c8cb0653a1e79fbf321f055d2cfdfd79a904b9c3a62b76213a5035"
Oct 30 13:45:07.441970 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.441943    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm_odh-ai-gateway-infra(380b156b-fe53-460f-aa79-2a9f98e37bd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2"
Oct 30 13:45:07.443972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.443955    4993 scope.go:122] "RemoveContainer" containerID="1580b838891b13f3aef1d83e4b4112aa1424a6e9c58436d602c5813c861a7887"
Oct 30 13:45:07.444183 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.444163    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm_odh-ai-gateway-infra(de1bfff9-9805-4b5a-a959-734a9a1570e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8"
Oct 30 13:45:07.445522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.445500    4993 generic.go:356] "Generic (PLEG): container finished" podID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerID="f312a3e4e55684ca0b5d3cb1fe1e38d5893e49feffde32fffc628410118f60b9" exitCode=60
Oct 30 13:45:07.445602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.445545    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" event={"ID":"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2","Type":"ContainerDied","Data":"f312a3e4e55684ca0b5d3cb1fe1e38d5893e49feffde32fffc628410118f60b9"}
Oct 30 13:45:07.445972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.445958    4993 scope.go:122] "RemoveContainer" containerID="f312a3e4e55684ca0b5d3cb1fe1e38d5893e49feffde32fffc628410118f60b9"
Oct 30 13:45:07.446276 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.446247    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh_odh-ai-gateway-infra(3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2"
Oct 30 13:45:07.451240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.451126    4993 scope.go:122] "RemoveContainer" containerID="649a526b330774c3cfe042b546f0863126946256e19d7af2c6004f9119c18717"
Oct 30 13:45:07.505401 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:07.504923    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de2b6ea_dd19_4d6f_8b8d_1f5155cd6cb2.slice/crio-conmon-f312a3e4e55684ca0b5d3cb1fe1e38d5893e49feffde32fffc628410118f60b9.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:45:07.606846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.606807    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"]
Oct 30 13:45:07.862888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.862837    4993 scope.go:122] "RemoveContainer" containerID="92bf01cf2a8e4236d6ce5ab0bd5000708d9c3279ff8e086dafe5774f59663297"
Oct 30 13:45:07.953628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.953600    4993 scope.go:122] "RemoveContainer" containerID="cc1873e73149fe45d6bbf3b23b84c9ff74927fbccd5f2190bd3922efd1c50d33"
Oct 30 13:45:07.978867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.978825    4993 scope.go:122] "RemoveContainer" containerID="8fe244a9bbfcd84fcdc7fe7405b0e89dc4ff993788e533512a2b4ff57049f3a8"
Oct 30 13:45:07.998319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:07.998283    4993 scope.go:122] "RemoveContainer" containerID="767ff54c76619e282ff816bb9819bd85ce7c25af87945e1eea3241807a6f828f"
Oct 30 13:45:08.015382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.015355    4993 scope.go:122] "RemoveContainer" containerID="1feff11508a7de9561e83ab2a7533423347af12d5f329a2be064b643a86c412d"
Oct 30 13:45:08.031727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.031682    4993 scope.go:122] "RemoveContainer" containerID="e6f38d8011abbdbe72e2f0d0bf3970af43a9f98fd491f657098ea31d5e4be771"
Oct 30 13:45:08.048028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.048001    4993 scope.go:122] "RemoveContainer" containerID="529b853b80f3056aafcd0502b205ac1de57e52ba37b9716dd8bf86bf92c74024"
Oct 30 13:45:08.132799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.132772    4993 scope.go:122] "RemoveContainer" containerID="63faa31eb09694a0c81e50c773f98aeca6f18726ba81b816f57c7f43d9bfd446"
Oct 30 13:45:08.244146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.244080    4993 scope.go:122] "RemoveContainer" containerID="d43c1c0f7395cd07e1f3031dc64f3efeed7e09c0f20467c0fb70311170c587eb"
Oct 30 13:45:08.459292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.459160    4993 scope.go:122] "RemoveContainer" containerID="b46b3bc85e0e5e334033f07d21e3214049ea617f9010d62d3642fd53fc2f7a95"
Oct 30 13:45:08.459655 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.459626    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj_odh-ai-gateway-infra(613f276e-cad4-40e0-84e8-b5c7e2524853)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853"
Oct 30 13:45:08.461128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.461098    4993 generic.go:356] "Generic (PLEG): container finished" podID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerID="aa02fab57371e233fff768d9f8fdafcf6e39139830edcf74738d8b7539591253" exitCode=60
Oct 30 13:45:08.461294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.461182    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" event={"ID":"75087b72-ac15-4d81-bd96-659ac0f7b2ef","Type":"ContainerDied","Data":"aa02fab57371e233fff768d9f8fdafcf6e39139830edcf74738d8b7539591253"}
Oct 30 13:45:08.461294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.461289    4993 scope.go:122] "RemoveContainer" containerID="c3a03f572d55378cdff05fad335136708d876e26f6a30485a857917c80257077"
Oct 30 13:45:08.461568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.461537    4993 scope.go:122] "RemoveContainer" containerID="aa02fab57371e233fff768d9f8fdafcf6e39139830edcf74738d8b7539591253"
Oct 30 13:45:08.461982 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.461958    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc_odh-ai-gateway-infra(75087b72-ac15-4d81-bd96-659ac0f7b2ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef"
Oct 30 13:45:08.463666 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.463649    4993 scope.go:122] "RemoveContainer" containerID="6e09601149486d8cdfad5aff658d9fed266fef0319d4684bb47f23e7c23ddb0e"
Oct 30 13:45:08.464040 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.464017    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4_odh-ai-gateway-infra(48a48c09-e2e1-4006-8d0c-2331997f7a85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85"
Oct 30 13:45:08.465697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.465672    4993 generic.go:356] "Generic (PLEG): container finished" podID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerID="a102c923dfa34b61c5fe96732c04b51e6c4ea2762f0f731f76fe5c8657ec870e" exitCode=60
Oct 30 13:45:08.465814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.465742    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" event={"ID":"a7bc62a2-0765-497f-8529-8a11acfeb6be","Type":"ContainerDied","Data":"a102c923dfa34b61c5fe96732c04b51e6c4ea2762f0f731f76fe5c8657ec870e"}
Oct 30 13:45:08.466035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.466013    4993 scope.go:122] "RemoveContainer" containerID="a102c923dfa34b61c5fe96732c04b51e6c4ea2762f0f731f76fe5c8657ec870e"
Oct 30 13:45:08.466422 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.466399    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb_odh-ai-gateway-infra(a7bc62a2-0765-497f-8529-8a11acfeb6be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be"
Oct 30 13:45:08.467941 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.467914    4993 generic.go:356] "Generic (PLEG): container finished" podID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerID="a607b50c5d6427b9b38c6f660674d9cdf682513f46cfef94440825753ecd7cfe" exitCode=60
Oct 30 13:45:08.468031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.467973    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" event={"ID":"552e7cb2-282d-4f7e-b4ca-952b9df7a041","Type":"ContainerDied","Data":"a607b50c5d6427b9b38c6f660674d9cdf682513f46cfef94440825753ecd7cfe"}
Oct 30 13:45:08.468321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.468300    4993 scope.go:122] "RemoveContainer" containerID="a607b50c5d6427b9b38c6f660674d9cdf682513f46cfef94440825753ecd7cfe"
Oct 30 13:45:08.468691 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.468670    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg_odh-ai-gateway-infra(552e7cb2-282d-4f7e-b4ca-952b9df7a041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041"
Oct 30 13:45:08.470749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.470727    4993 scope.go:122] "RemoveContainer" containerID="a97a40cd0d53f263564d082e6216291d65151270af5ea8640114cc2ab928c8e5"
Oct 30 13:45:08.471307 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.471281    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh_odh-ai-gateway-infra(816d95a5-15ed-418c-93f1-79b39600a5e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3"
Oct 30 13:45:08.473063 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.473039    4993 generic.go:356] "Generic (PLEG): container finished" podID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerID="f8835816482613c783d7fae8974012eb1c8e002b63f46b16cccc5a17ba0bad27" exitCode=60
Oct 30 13:45:08.473181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.473125    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" event={"ID":"9325a541-f5aa-4272-bbe1-c75fd95e384a","Type":"ContainerDied","Data":"f8835816482613c783d7fae8974012eb1c8e002b63f46b16cccc5a17ba0bad27"}
Oct 30 13:45:08.473434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.473412    4993 scope.go:122] "RemoveContainer" containerID="f8835816482613c783d7fae8974012eb1c8e002b63f46b16cccc5a17ba0bad27"
Oct 30 13:45:08.473782 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.473757    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8_odh-ai-gateway-infra(9325a541-f5aa-4272-bbe1-c75fd95e384a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a"
Oct 30 13:45:08.475368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.475345    4993 generic.go:356] "Generic (PLEG): container finished" podID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerID="b5b00b578b98f5f666a082a67f37ea1b3353cb22a89843d039f187ab0448a11f" exitCode=60
Oct 30 13:45:08.475496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.475440    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" event={"ID":"d8394c36-ec95-44a0-a6d3-a01716310ebc","Type":"ContainerDied","Data":"b5b00b578b98f5f666a082a67f37ea1b3353cb22a89843d039f187ab0448a11f"}
Oct 30 13:45:08.475730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.475711    4993 scope.go:122] "RemoveContainer" containerID="b5b00b578b98f5f666a082a67f37ea1b3353cb22a89843d039f187ab0448a11f"
Oct 30 13:45:08.476053 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.476032    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2_odh-ai-gateway-infra(d8394c36-ec95-44a0-a6d3-a01716310ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc"
Oct 30 13:45:08.477899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.477877    4993 scope.go:122] "RemoveContainer" containerID="5b817eba7ecac6613ced24bcb35dd54101c82d26bacfde1a5e69ebb419612331"
Oct 30 13:45:08.478187 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.478165    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb_odh-ai-gateway-infra(a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0"
Oct 30 13:45:08.481623 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.481606    4993 scope.go:122] "RemoveContainer" containerID="e49d928eb247a3398af6c9269bf618c86b308b216c2f4185278e95fbd05774fa"
Oct 30 13:45:08.481753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.481737    4993 generic.go:356] "Generic (PLEG): container finished" podID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerID="9f2673c69992922314806e9f21edd9c2ffe5fc1d933338763f0607ebb90a3c2c" exitCode=60
Oct 30 13:45:08.481839 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.481814    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" event={"ID":"7febc0c4-d800-4a15-b9b8-708d4517a176","Type":"ContainerDied","Data":"9f2673c69992922314806e9f21edd9c2ffe5fc1d933338763f0607ebb90a3c2c"}
Oct 30 13:45:08.482184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.482164    4993 scope.go:122] "RemoveContainer" containerID="9f2673c69992922314806e9f21edd9c2ffe5fc1d933338763f0607ebb90a3c2c"
Oct 30 13:45:08.482563 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.482470    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz_odh-ai-gateway-infra(7febc0c4-d800-4a15-b9b8-708d4517a176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176"
Oct 30 13:45:08.484066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.484048    4993 scope.go:122] "RemoveContainer" containerID="d86bab65fba7dd522607859003bced0e3894cc12c424b887a1ff2d416759990e"
Oct 30 13:45:08.484353 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.484325    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8_odh-ai-gateway-infra(7f0393d0-1fa2-4248-9d19-f31b08556afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa"
Oct 30 13:45:08.485907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.485888    4993 scope.go:122] "RemoveContainer" containerID="1748aaf25bb7978775531c0b742434109882d2457e718cfb933791a52102d0a9"
Oct 30 13:45:08.486136 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.486115    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8_odh-ai-gateway-infra(2907709a-ad1d-4724-ab42-02ad7defb4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1"
Oct 30 13:45:08.487750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.487731    4993 scope.go:122] "RemoveContainer" containerID="18255b0eb7c8cb0653a1e79fbf321f055d2cfdfd79a904b9c3a62b76213a5035"
Oct 30 13:45:08.487978 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.487959    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm_odh-ai-gateway-infra(380b156b-fe53-460f-aa79-2a9f98e37bd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2"
Oct 30 13:45:08.489815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.489793    4993 scope.go:122] "RemoveContainer" containerID="f312a3e4e55684ca0b5d3cb1fe1e38d5893e49feffde32fffc628410118f60b9"
Oct 30 13:45:08.490071 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.490047    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh_odh-ai-gateway-infra(3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2"
Oct 30 13:45:08.491917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.491893    4993 generic.go:356] "Generic (PLEG): container finished" podID="49589327-9884-4386-a522-df5b12d23094" containerID="093d43232dde3539ecb5a71e0daa7382f8ca1db68788b212819795224c73c74f" exitCode=60
Oct 30 13:45:08.492034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.491987    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" event={"ID":"49589327-9884-4386-a522-df5b12d23094","Type":"ContainerDied","Data":"093d43232dde3539ecb5a71e0daa7382f8ca1db68788b212819795224c73c74f"}
Oct 30 13:45:08.492224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.492182    4993 scope.go:122] "RemoveContainer" containerID="093d43232dde3539ecb5a71e0daa7382f8ca1db68788b212819795224c73c74f"
Oct 30 13:45:08.492480 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.492457    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg_odh-ai-gateway-infra(49589327-9884-4386-a522-df5b12d23094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" podUID="49589327-9884-4386-a522-df5b12d23094"
Oct 30 13:45:08.494411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.494387    4993 scope.go:122] "RemoveContainer" containerID="dce0d8a607e4a4df82f79c1964ef9d7575f82a432a17f4edc17d10821026efc1"
Oct 30 13:45:08.494669 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.494645    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8_odh-ai-gateway-infra(b96952ed-4208-4568-9005-6a53e3f36eb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6"
Oct 30 13:45:08.496433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.496400    4993 generic.go:356] "Generic (PLEG): container finished" podID="66456109-7df3-4de7-978e-13496801fff2" containerID="0ebe92f8ca0985c2da7da6f7671ff023bbbb9794524d8b0513a5cc15dd510221" exitCode=60
Oct 30 13:45:08.496527 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.496494    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" event={"ID":"66456109-7df3-4de7-978e-13496801fff2","Type":"ContainerDied","Data":"0ebe92f8ca0985c2da7da6f7671ff023bbbb9794524d8b0513a5cc15dd510221"}
Oct 30 13:45:08.496843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.496822    4993 scope.go:122] "RemoveContainer" containerID="0ebe92f8ca0985c2da7da6f7671ff023bbbb9794524d8b0513a5cc15dd510221"
Oct 30 13:45:08.497106 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.497083    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h_odh-ai-gateway-infra(66456109-7df3-4de7-978e-13496801fff2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" podUID="66456109-7df3-4de7-978e-13496801fff2"
Oct 30 13:45:08.499597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.499576    4993 scope.go:122] "RemoveContainer" containerID="7f5a1b3af0c71acd1b09395bfff5d01b4822c0bfb07566dc171bf79c4a8564e8"
Oct 30 13:45:08.502473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.502454    4993 scope.go:122] "RemoveContainer" containerID="835117598a783b80b0f477b7316de8825b438e2e5117cd1487b3f585bcc4a3e9"
Oct 30 13:45:08.502838 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.502800    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9_odh-ai-gateway-infra(998580a2-a1bf-4eb6-b8db-b3277af46957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957"
Oct 30 13:45:08.504827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.504805    4993 scope.go:122] "RemoveContainer" containerID="3a59798e0578c8bc91ef567f653b0d1201a3027734377aacf80ca36ebe2e4806"
Oct 30 13:45:08.505108 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.505083    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp_odh-ai-gateway-infra(59ac7f13-0f86-43e3-9189-d89aa39defb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9"
Oct 30 13:45:08.507157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.507136    4993 generic.go:356] "Generic (PLEG): container finished" podID="f49c1907-0584-44be-8022-3b27cc76ec35" containerID="3eaa7c05312493274a1734abe92bc088e7328d15bc3c07734d3ea7543ed3f9e3" exitCode=60
Oct 30 13:45:08.507271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.507234    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" event={"ID":"f49c1907-0584-44be-8022-3b27cc76ec35","Type":"ContainerDied","Data":"3eaa7c05312493274a1734abe92bc088e7328d15bc3c07734d3ea7543ed3f9e3"}
Oct 30 13:45:08.507506 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.507489    4993 scope.go:122] "RemoveContainer" containerID="3eaa7c05312493274a1734abe92bc088e7328d15bc3c07734d3ea7543ed3f9e3"
Oct 30 13:45:08.507856 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.507755    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2_odh-ai-gateway-infra(f49c1907-0584-44be-8022-3b27cc76ec35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" podUID="f49c1907-0584-44be-8022-3b27cc76ec35"
Oct 30 13:45:08.509387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.509369    4993 scope.go:122] "RemoveContainer" containerID="94e714e37b94a4d1afd7e20af0efe6369d099001e46b1b395764e9e1c6b590b2"
Oct 30 13:45:08.509621 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.509598    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw_odh-ai-gateway-infra(ebf48e8e-316b-4250-9bae-9bcb736d3cb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9"
Oct 30 13:45:08.517328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.517306    4993 scope.go:122] "RemoveContainer" containerID="3b09b6577f26d6a4b54ef67e904f2d647db53e3f30acdbb7f8679913faeebf4d"
Oct 30 13:45:08.532657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.532628    4993 scope.go:122] "RemoveContainer" containerID="086d900c6e02d53d98cc09c35c0d0dd1671d97c28490c50141d3567a23c0cbda"
Oct 30 13:45:08.550880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.550861    4993 scope.go:122] "RemoveContainer" containerID="72fab3a96379f8d6010ba4560963eed10074cf550d634585130896a2360762b7"
Oct 30 13:45:08.567728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.567689    4993 scope.go:122] "RemoveContainer" containerID="9ad42ee23c53f97dc1c95bf24ecd9034b53c98e62df002708cec2459f904737e"
Oct 30 13:45:08.573978 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:08.573944    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:45:08.588938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.588388    4993 scope.go:122] "RemoveContainer" containerID="9a810f0776e70ead9f8608abc0c67ec8cd983b7805db463e66713a5472a6527b"
Oct 30 13:45:08.611125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:08.611083    4993 scope.go:122] "RemoveContainer" containerID="1f0df0c136ece1c917d124d242f1cdb482859d3f9912b59f68f148833bb8d4b3"
Oct 30 13:45:09.523036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:09.522856    4993 scope.go:122] "RemoveContainer" containerID="093d43232dde3539ecb5a71e0daa7382f8ca1db68788b212819795224c73c74f"
Oct 30 13:45:09.523590 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:09.523391    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg_odh-ai-gateway-infra(49589327-9884-4386-a522-df5b12d23094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" podUID="49589327-9884-4386-a522-df5b12d23094"
Oct 30 13:45:09.525275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:09.525239    4993 scope.go:122] "RemoveContainer" containerID="0ebe92f8ca0985c2da7da6f7671ff023bbbb9794524d8b0513a5cc15dd510221"
Oct 30 13:45:09.525645 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:09.525621    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h_odh-ai-gateway-infra(66456109-7df3-4de7-978e-13496801fff2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" podUID="66456109-7df3-4de7-978e-13496801fff2"
Oct 30 13:45:09.526906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:09.526884    4993 scope.go:122] "RemoveContainer" containerID="3eaa7c05312493274a1734abe92bc088e7328d15bc3c07734d3ea7543ed3f9e3"
Oct 30 13:45:09.527248 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:09.527222    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2_odh-ai-gateway-infra(f49c1907-0584-44be-8022-3b27cc76ec35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" podUID="f49c1907-0584-44be-8022-3b27cc76ec35"
Oct 30 13:45:09.528590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:09.528560    4993 scope.go:122] "RemoveContainer" containerID="aa02fab57371e233fff768d9f8fdafcf6e39139830edcf74738d8b7539591253"
Oct 30 13:45:09.528933 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:09.528910    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc_odh-ai-gateway-infra(75087b72-ac15-4d81-bd96-659ac0f7b2ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef"
Oct 30 13:45:09.530132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:09.530114    4993 scope.go:122] "RemoveContainer" containerID="a102c923dfa34b61c5fe96732c04b51e6c4ea2762f0f731f76fe5c8657ec870e"
Oct 30 13:45:09.530392 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:09.530372    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb_odh-ai-gateway-infra(a7bc62a2-0765-497f-8529-8a11acfeb6be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be"
Oct 30 13:45:09.531806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:09.531784    4993 scope.go:122] "RemoveContainer" containerID="a607b50c5d6427b9b38c6f660674d9cdf682513f46cfef94440825753ecd7cfe"
Oct 30 13:45:09.532120 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:09.532095    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg_odh-ai-gateway-infra(552e7cb2-282d-4f7e-b4ca-952b9df7a041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041"
Oct 30 13:45:09.533534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:09.533514    4993 scope.go:122] "RemoveContainer" containerID="f8835816482613c783d7fae8974012eb1c8e002b63f46b16cccc5a17ba0bad27"
Oct 30 13:45:09.533787 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:09.533766    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8_odh-ai-gateway-infra(9325a541-f5aa-4272-bbe1-c75fd95e384a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a"
Oct 30 13:45:09.535171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:09.535154    4993 scope.go:122] "RemoveContainer" containerID="b5b00b578b98f5f666a082a67f37ea1b3353cb22a89843d039f187ab0448a11f"
Oct 30 13:45:09.535415 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:09.535389    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2_odh-ai-gateway-infra(d8394c36-ec95-44a0-a6d3-a01716310ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc"
Oct 30 13:45:14.918973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:14.918906    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"]
Oct 30 13:45:15.977289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:15.970119    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"]
Oct 30 13:45:16.168229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.164755    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22efcf-pfjrm"]
Oct 30 13:45:16.407350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.390282    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:16.631232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.625495    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a48c09-e2e1-4006-8d0c-2331997f7a85-kube-api-access-l6t2f\" (UniqueName: \"kubernetes.io/projected/48a48c09-e2e1-4006-8d0c-2331997f7a85-kube-api-access-l6t2f\") pod \"48a48c09-e2e1-4006-8d0c-2331997f7a85\" (UID: \"48a48c09-e2e1-4006-8d0c-2331997f7a85\") "
Oct 30 13:45:16.649900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.649851    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a48c09-e2e1-4006-8d0c-2331997f7a85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a48c09-e2e1-4006-8d0c-2331997f7a85-maas-api-service-ca\") pod \"48a48c09-e2e1-4006-8d0c-2331997f7a85\" (UID: \"48a48c09-e2e1-4006-8d0c-2331997f7a85\") "
Oct 30 13:45:16.730285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.696468    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a48c09-e2e1-4006-8d0c-2331997f7a85-maas-api-service-ca" pod "48a48c09-e2e1-4006-8d0c-2331997f7a85" (UID: "48a48c09-e2e1-4006-8d0c-2331997f7a85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:16.730285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.720547    4993 scope.go:122] "RemoveContainer" containerID="8791be731685325c996fb68ef2c606aa1119745ab3c5ef70bb7422d21bbfc57b"
Oct 30 13:45:16.748906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.748870    4993 scope.go:122] "RemoveContainer" containerID="1580b838891b13f3aef1d83e4b4112aa1424a6e9c58436d602c5813c861a7887"
Oct 30 13:45:16.757782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.757624    4993 scope.go:122] "RemoveContainer" containerID="dce0d8a607e4a4df82f79c1964ef9d7575f82a432a17f4edc17d10821026efc1"
Oct 30 13:45:16.759331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.758891    4993 scope.go:122] "RemoveContainer" containerID="f312a3e4e55684ca0b5d3cb1fe1e38d5893e49feffde32fffc628410118f60b9"
Oct 30 13:45:16.776010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:16.775955    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a48c09-e2e1-4006-8d0c-2331997f7a85-kube-api-access-l6t2f" pod "48a48c09-e2e1-4006-8d0c-2331997f7a85" (UID: "48a48c09-e2e1-4006-8d0c-2331997f7a85"). InnerVolumeSpecName "kube-api-access-l6t2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:17.218230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.217403    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 13:45:17.265259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.220074    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk" podUID="5bf8bf0b-166c-44c4-be3e-0b8488950bb3" containerName="main" probeResult="failure" output="Get \"https://10.132.1.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:17.265686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.265569    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a48c09-e2e1-4006-8d0c-2331997f7a85-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:17.265686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.265599    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6t2f\" (UniqueName: \"kubernetes.io/projected/48a48c09-e2e1-4006-8d0c-2331997f7a85-kube-api-access-l6t2f\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:17.297267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.297226    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:45:17.334268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.298596    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:17.381264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.351323    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:17.409533 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.409476    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" podUID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:17.559242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.517772    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4" event={"ID":"48a48c09-e2e1-4006-8d0c-2331997f7a85","Type":"ContainerDied","Data":"c52ce7a84ec1e03677e0e165662c5a3c3a145a0bfd67a06c396284e5b4f6a929"}
Oct 30 13:45:17.559242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.517858    4993 scope.go:122] "RemoveContainer" containerID="6e09601149486d8cdfad5aff658d9fed266fef0319d4684bb47f23e7c23ddb0e"
Oct 30 13:45:17.559242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.518033    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"
Oct 30 13:45:17.790593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.790545    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:17.903349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:17.903305    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:18.039376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.037390    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2d2426-ac66-4791-9152-032cad4046c3" path="/var/lib/kubelet/pods/ba2d2426-ac66-4791-9152-032cad4046c3/volumes"
Oct 30 13:45:18.061249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.044119    4993 scope.go:122] "RemoveContainer" containerID="94e714e37b94a4d1afd7e20af0efe6369d099001e46b1b395764e9e1c6b590b2"
Oct 30 13:45:18.062004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.061970    4993 scope.go:122] "RemoveContainer" containerID="aa02fab57371e233fff768d9f8fdafcf6e39139830edcf74738d8b7539591253"
Oct 30 13:45:18.067073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.067048    4993 scope.go:122] "RemoveContainer" containerID="b46b3bc85e0e5e334033f07d21e3214049ea617f9010d62d3642fd53fc2f7a95"
Oct 30 13:45:18.141317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.129960    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:18.141317 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:18.135349    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a48c09_e2e1_4006_8d0c_2331997f7a85.slice/crio-c52ce7a84ec1e03677e0e165662c5a3c3a145a0bfd67a06c396284e5b4f6a929\": RecentStats: unable to find data in memory cache]"
Oct 30 13:45:18.277028 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:18.253033    4993 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a48c09_e2e1_4006_8d0c_2331997f7a85.slice/crio-c52ce7a84ec1e03677e0e165662c5a3c3a145a0bfd67a06c396284e5b4f6a929\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a48c09_e2e1_4006_8d0c_2331997f7a85.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:45:18.277028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.272935    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:18.345231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.335753    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:18.345231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.337929    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"]
Oct 30 13:45:18.372358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.372303    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:18.430280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.419389    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22efcf-29889465-z77t4"]
Oct 30 13:45:18.564240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.556405    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:18.725231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.705270    4993 scope.go:122] "RemoveContainer" containerID="b5b00b578b98f5f666a082a67f37ea1b3353cb22a89843d039f187ab0448a11f"
Oct 30 13:45:18.725231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.706229    4993 scope.go:122] "RemoveContainer" containerID="5b817eba7ecac6613ced24bcb35dd54101c82d26bacfde1a5e69ebb419612331"
Oct 30 13:45:18.725231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.711696    4993 scope.go:122] "RemoveContainer" containerID="18255b0eb7c8cb0653a1e79fbf321f055d2cfdfd79a904b9c3a62b76213a5035"
Oct 30 13:45:18.725231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.712600    4993 scope.go:122] "RemoveContainer" containerID="0ebe92f8ca0985c2da7da6f7671ff023bbbb9794524d8b0513a5cc15dd510221"
Oct 30 13:45:18.725231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.713924    4993 scope.go:122] "RemoveContainer" containerID="3345d718bb7bd8aec7d0a9ec5be1d09d98e5130e9559fd900342539c04505a24"
Oct 30 13:45:18.725231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.714603    4993 scope.go:122] "RemoveContainer" containerID="a97a40cd0d53f263564d082e6216291d65151270af5ea8640114cc2ab928c8e5"
Oct 30 13:45:18.728531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.728505    4993 scope.go:122] "RemoveContainer" containerID="a102c923dfa34b61c5fe96732c04b51e6c4ea2762f0f731f76fe5c8657ec870e"
Oct 30 13:45:18.828252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.828105    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" event={"ID":"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6","Type":"ContainerStarted","Data":"a1e030ce85cf57219640310dff9dc580c5e18d1ae2e1a1defc9de244ebe9dda1"}
Oct 30 13:45:18.909391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.908778    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:18.909391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:18.909236    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:45:19.251295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.084434    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:45:19.251295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.132774    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:19.251295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.133374    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 13:45:19.251295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.136518    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.54:8000/ready\": context deadline exceeded"
Oct 30 13:45:19.309947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.284503    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:19.309947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.306428    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 13:45:19.359267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.325993    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:19.575092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.574173    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:45:19.575092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.574592    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Oct 30 13:45:19.575612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.575595    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:45:19.639982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.575870    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:19.705233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.680684    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 13:45:19.960238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.960192    4993 scope.go:122] "RemoveContainer" containerID="835117598a783b80b0f477b7316de8825b438e2e5117cd1487b3f585bcc4a3e9"
Oct 30 13:45:19.972559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.972520    4993 scope.go:122] "RemoveContainer" containerID="1748aaf25bb7978775531c0b742434109882d2457e718cfb933791a52102d0a9"
Oct 30 13:45:19.974521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.974500    4993 scope.go:122] "RemoveContainer" containerID="3a59798e0578c8bc91ef567f653b0d1201a3027734377aacf80ca36ebe2e4806"
Oct 30 13:45:19.980142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.979745    4993 scope.go:122] "RemoveContainer" containerID="d86bab65fba7dd522607859003bced0e3894cc12c424b887a1ff2d416759990e"
Oct 30 13:45:19.982289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:19.982269    4993 scope.go:122] "RemoveContainer" containerID="3eaa7c05312493274a1734abe92bc088e7328d15bc3c07734d3ea7543ed3f9e3"
Oct 30 13:45:20.553574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:20.548435    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:20.600921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:20.582835    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:20.639229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:20.606062    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" podUID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.52:8000/ready\": context deadline exceeded"
Oct 30 13:45:20.837365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:20.700707    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:21.337325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.333932    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85" path="/var/lib/kubelet/pods/48a48c09-e2e1-4006-8d0c-2331997f7a85/volumes"
Oct 30 13:45:21.341550 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:21.341512    4993 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.398s"
Oct 30 13:45:21.342295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.342119    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 13:45:21.342540 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.342521    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 13:45:21.342894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.342660    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"]
Oct 30 13:45:21.344400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.344266    4993 scope.go:122] "RemoveContainer" containerID="a607b50c5d6427b9b38c6f660674d9cdf682513f46cfef94440825753ecd7cfe"
Oct 30 13:45:21.344400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.344318    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 13:45:21.344400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.344365    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:45:21.344400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.344384    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"]
Oct 30 13:45:21.346395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.346158    4993 scope.go:122] "RemoveContainer" containerID="093d43232dde3539ecb5a71e0daa7382f8ca1db68788b212819795224c73c74f"
Oct 30 13:45:21.845299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.721089    4993 scope.go:122] "RemoveContainer" containerID="f8835816482613c783d7fae8974012eb1c8e002b63f46b16cccc5a17ba0bad27"
Oct 30 13:45:21.845299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:21.722151    4993 scope.go:122] "RemoveContainer" containerID="9f2673c69992922314806e9f21edd9c2ffe5fc1d933338763f0607ebb90a3c2c"
Oct 30 13:45:22.027777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.027739    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" event={"ID":"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2","Type":"ContainerStarted","Data":"316fdbe5d10c6fff8abb99172b47e73d8ecf0e347f01fa81703e9ff1be2956df"}
Oct 30 13:45:22.028847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.028805    4993 scope.go:122] "RemoveContainer" containerID="316fdbe5d10c6fff8abb99172b47e73d8ecf0e347f01fa81703e9ff1be2956df"
Oct 30 13:45:22.030435 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:22.030406    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh_odh-ai-gateway-infra(3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2"
Oct 30 13:45:22.087765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.037736    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" event={"ID":"b96952ed-4208-4568-9005-6a53e3f36eb6","Type":"ContainerStarted","Data":"947e0a22542b20d0accb64c0e849508dd920efff7325a0ed033802b17cab046c"}
Oct 30 13:45:22.122795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.121684    4993 generic.go:356] "Generic (PLEG): container finished" podID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerID="a1e030ce85cf57219640310dff9dc580c5e18d1ae2e1a1defc9de244ebe9dda1" exitCode=60
Oct 30 13:45:22.122795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.121769    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" event={"ID":"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6","Type":"ContainerDied","Data":"a1e030ce85cf57219640310dff9dc580c5e18d1ae2e1a1defc9de244ebe9dda1"}
Oct 30 13:45:22.122795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.121827    4993 scope.go:122] "RemoveContainer" containerID="8791be731685325c996fb68ef2c606aa1119745ab3c5ef70bb7422d21bbfc57b"
Oct 30 13:45:22.176826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.174671    4993 scope.go:122] "RemoveContainer" containerID="947e0a22542b20d0accb64c0e849508dd920efff7325a0ed033802b17cab046c"
Oct 30 13:45:22.176826 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:22.175003    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8_odh-ai-gateway-infra(b96952ed-4208-4568-9005-6a53e3f36eb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6"
Oct 30 13:45:22.418018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.417930    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk" podUID="5bf8bf0b-166c-44c4-be3e-0b8488950bb3" containerName="main" containerID="cri-o://bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e" gracePeriod=45
Oct 30 13:45:22.641615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.641564    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" podUID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" containerName="main" containerID="cri-o://62160013bd807ef3f6f80c0673e77b55b7dae018bd2e157bcb216373c8d58bba" gracePeriod=45
Oct 30 13:45:22.898913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.898490    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:22.934753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.934112    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-kube-api-access-8qpft\" (UniqueName: \"kubernetes.io/projected/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-kube-api-access-8qpft\") pod \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\" (UID: \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\") "
Oct 30 13:45:22.935298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.935270    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-maas-api-service-ca\") pod \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\" (UID: \"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6\") "
Oct 30 13:45:22.937742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.935776    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-maas-api-service-ca" pod "8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" (UID: "8e7f69c1-5545-4e9c-8aee-2046fb00d5c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:22.937742 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.936917    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:22.941397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:22.941348    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-kube-api-access-8qpft" pod "8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" (UID: "8e7f69c1-5545-4e9c-8aee-2046fb00d5c6"). InnerVolumeSpecName "kube-api-access-8qpft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.039866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.039827    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpft\" (UniqueName: \"kubernetes.io/projected/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6-kube-api-access-8qpft\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.132496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.132448    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"]
Oct 30 13:45:23.164149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.164112    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:45:23.198861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.197386    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"]
Oct 30 13:45:23.244762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.244725    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-model-cache\") pod \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") "
Oct 30 13:45:23.244967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.244831    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tls-certs\") pod \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") "
Oct 30 13:45:23.244967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.244858    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tmp-dir\") pod \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") "
Oct 30 13:45:23.244967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.244936    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-kube-api-access-77pjq\" (UniqueName: \"kubernetes.io/projected/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-kube-api-access-77pjq\") pod \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") "
Oct 30 13:45:23.245135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.245036    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-model-cache" pod "5bf8bf0b-166c-44c4-be3e-0b8488950bb3" (UID: "5bf8bf0b-166c-44c4-be3e-0b8488950bb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:23.245135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.245104    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-dshm\") pod \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") "
Oct 30 13:45:23.245759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.245196    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-home\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-home\") pod \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\" (UID: \"5bf8bf0b-166c-44c4-be3e-0b8488950bb3\") "
Oct 30 13:45:23.245759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.245332    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tmp-dir" pod "5bf8bf0b-166c-44c4-be3e-0b8488950bb3" (UID: "5bf8bf0b-166c-44c4-be3e-0b8488950bb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:23.246072 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.246050    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.246166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.246078    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.246349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.246329    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-home" pod "5bf8bf0b-166c-44c4-be3e-0b8488950bb3" (UID: "5bf8bf0b-166c-44c4-be3e-0b8488950bb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:23.263710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.262358    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tls-certs" pod "5bf8bf0b-166c-44c4-be3e-0b8488950bb3" (UID: "5bf8bf0b-166c-44c4-be3e-0b8488950bb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:23.263710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.262784    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-dshm" pod "5bf8bf0b-166c-44c4-be3e-0b8488950bb3" (UID: "5bf8bf0b-166c-44c4-be3e-0b8488950bb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:23.263710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.263282    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-kube-api-access-77pjq" pod "5bf8bf0b-166c-44c4-be3e-0b8488950bb3" (UID: "5bf8bf0b-166c-44c4-be3e-0b8488950bb3"). InnerVolumeSpecName "kube-api-access-77pjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.347573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.347533    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.347573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.347565    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.347573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.347575    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.348512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.347586    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77pjq\" (UniqueName: \"kubernetes.io/projected/5bf8bf0b-166c-44c4-be3e-0b8488950bb3-kube-api-access-77pjq\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.417046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.417013    4993 generic.go:356] "Generic (PLEG): container finished" podID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" containerID="62160013bd807ef3f6f80c0673e77b55b7dae018bd2e157bcb216373c8d58bba" exitCode=0
Oct 30 13:45:23.417181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.417120    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" event={"ID":"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b","Type":"ContainerDied","Data":"62160013bd807ef3f6f80c0673e77b55b7dae018bd2e157bcb216373c8d58bba"}
Oct 30 13:45:23.417181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.417158    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" event={"ID":"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b","Type":"ContainerDied","Data":"d2af08effd1873dcdc1fe337b3d287fea5e085ee954d9b412309b5443a0f5330"}
Oct 30 13:45:23.417181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.417175    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2af08effd1873dcdc1fe337b3d287fea5e085ee954d9b412309b5443a0f5330"
Oct 30 13:45:23.419502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.419287    4993 generic.go:356] "Generic (PLEG): container finished" podID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerID="947e0a22542b20d0accb64c0e849508dd920efff7325a0ed033802b17cab046c" exitCode=60
Oct 30 13:45:23.419502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.419370    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" event={"ID":"b96952ed-4208-4568-9005-6a53e3f36eb6","Type":"ContainerDied","Data":"947e0a22542b20d0accb64c0e849508dd920efff7325a0ed033802b17cab046c"}
Oct 30 13:45:23.419502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.419428    4993 scope.go:122] "RemoveContainer" containerID="dce0d8a607e4a4df82f79c1964ef9d7575f82a432a17f4edc17d10821026efc1"
Oct 30 13:45:23.422047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.422023    4993 generic.go:356] "Generic (PLEG): container finished" podID="66456109-7df3-4de7-978e-13496801fff2" containerID="7e1c9c5cb7ce53c265471e99abcb2a195233b4a19f8ac355879a3aba9a76ae05" exitCode=60
Oct 30 13:45:23.422147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.422098    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" event={"ID":"66456109-7df3-4de7-978e-13496801fff2","Type":"ContainerDied","Data":"7e1c9c5cb7ce53c265471e99abcb2a195233b4a19f8ac355879a3aba9a76ae05"}
Oct 30 13:45:23.422565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.422548    4993 scope.go:122] "RemoveContainer" containerID="7e1c9c5cb7ce53c265471e99abcb2a195233b4a19f8ac355879a3aba9a76ae05"
Oct 30 13:45:23.423411 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.423082    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h_odh-ai-gateway-infra(66456109-7df3-4de7-978e-13496801fff2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" podUID="66456109-7df3-4de7-978e-13496801fff2"
Oct 30 13:45:23.424730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.424708    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" event={"ID":"998580a2-a1bf-4eb6-b8db-b3277af46957","Type":"ContainerStarted","Data":"f050d6f3c203a8c04522da8ee82cdedffce701fd7d68150404e5edc8470da2e0"}
Oct 30 13:45:23.427326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.426980    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" event={"ID":"59ac7f13-0f86-43e3-9189-d89aa39defb9","Type":"ContainerStarted","Data":"f63269caea7adbef39ee6b58d02c56f3fddaf11cc1ba84cdeb3718bc46fa2fa3"}
Oct 30 13:45:23.429329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.429305    4993 generic.go:356] "Generic (PLEG): container finished" podID="f49c1907-0584-44be-8022-3b27cc76ec35" containerID="6dc103bd0a942386317fd7e5ac2ab665824323bdcaa57901a6542a1ef887b632" exitCode=60
Oct 30 13:45:23.429421 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.429380    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" event={"ID":"f49c1907-0584-44be-8022-3b27cc76ec35","Type":"ContainerDied","Data":"6dc103bd0a942386317fd7e5ac2ab665824323bdcaa57901a6542a1ef887b632"}
Oct 30 13:45:23.429815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.429796    4993 scope.go:122] "RemoveContainer" containerID="6dc103bd0a942386317fd7e5ac2ab665824323bdcaa57901a6542a1ef887b632"
Oct 30 13:45:23.430217 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.430178    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2_odh-ai-gateway-infra(f49c1907-0584-44be-8022-3b27cc76ec35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" podUID="f49c1907-0584-44be-8022-3b27cc76ec35"
Oct 30 13:45:23.430976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.430952    4993 generic.go:356] "Generic (PLEG): container finished" podID="5bf8bf0b-166c-44c4-be3e-0b8488950bb3" containerID="bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e" exitCode=0
Oct 30 13:45:23.431088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.430988    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"
Oct 30 13:45:23.431088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.431038    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk" event={"ID":"5bf8bf0b-166c-44c4-be3e-0b8488950bb3","Type":"ContainerDied","Data":"bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e"}
Oct 30 13:45:23.431088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.431066    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk" event={"ID":"5bf8bf0b-166c-44c4-be3e-0b8488950bb3","Type":"ContainerDied","Data":"2bc63b75e509fa0a02b0aab552e6d011ed84bdd7cef0b3460838d6f26af0ce3e"}
Oct 30 13:45:23.432793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.432773    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"
Oct 30 13:45:23.432793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.432782    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782" event={"ID":"8e7f69c1-5545-4e9c-8aee-2046fb00d5c6","Type":"ContainerDied","Data":"e868d47b6e323c37c2ff03b9c3cf30d2578452bc93755950da34d5218ee5dce4"}
Oct 30 13:45:23.435852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.435824    4993 generic.go:356] "Generic (PLEG): container finished" podID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerID="f5e219312b6331e77e4c4d1993181d9fdb208bd0438ab9de8dd115ea0844b2d1" exitCode=60
Oct 30 13:45:23.435852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.435845    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" event={"ID":"ebf48e8e-316b-4250-9bae-9bcb736d3cb9","Type":"ContainerDied","Data":"f5e219312b6331e77e4c4d1993181d9fdb208bd0438ab9de8dd115ea0844b2d1"}
Oct 30 13:45:23.436356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.436335    4993 scope.go:122] "RemoveContainer" containerID="f5e219312b6331e77e4c4d1993181d9fdb208bd0438ab9de8dd115ea0844b2d1"
Oct 30 13:45:23.436711 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.436686    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw_odh-ai-gateway-infra(ebf48e8e-316b-4250-9bae-9bcb736d3cb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9"
Oct 30 13:45:23.438027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.438003    4993 generic.go:356] "Generic (PLEG): container finished" podID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerID="3f2ed0a405be7c795777799aaa58fb5733b815617e9643856111f9554b8e6256" exitCode=60
Oct 30 13:45:23.438136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.438066    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" event={"ID":"613f276e-cad4-40e0-84e8-b5c7e2524853","Type":"ContainerDied","Data":"3f2ed0a405be7c795777799aaa58fb5733b815617e9643856111f9554b8e6256"}
Oct 30 13:45:23.439317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.439292    4993 scope.go:122] "RemoveContainer" containerID="3f2ed0a405be7c795777799aaa58fb5733b815617e9643856111f9554b8e6256"
Oct 30 13:45:23.439705 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.439683    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj_odh-ai-gateway-infra(613f276e-cad4-40e0-84e8-b5c7e2524853)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853"
Oct 30 13:45:23.441802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.441781    4993 generic.go:356] "Generic (PLEG): container finished" podID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerID="3a6c19da578c1987c97b78c3dd2791dc569ef799141f2ed38e2659f437f40057" exitCode=60
Oct 30 13:45:23.441884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.441868    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" event={"ID":"75087b72-ac15-4d81-bd96-659ac0f7b2ef","Type":"ContainerDied","Data":"3a6c19da578c1987c97b78c3dd2791dc569ef799141f2ed38e2659f437f40057"}
Oct 30 13:45:23.442298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.442279    4993 scope.go:122] "RemoveContainer" containerID="3a6c19da578c1987c97b78c3dd2791dc569ef799141f2ed38e2659f437f40057"
Oct 30 13:45:23.442621 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.442598    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc_odh-ai-gateway-infra(75087b72-ac15-4d81-bd96-659ac0f7b2ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef"
Oct 30 13:45:23.444196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.444174    4993 generic.go:356] "Generic (PLEG): container finished" podID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerID="cb111761756dcab4012b532710867d4316e7c26090b495fd3a17077415501ee6" exitCode=60
Oct 30 13:45:23.444324 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.444259    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" event={"ID":"a7bc62a2-0765-497f-8529-8a11acfeb6be","Type":"ContainerDied","Data":"cb111761756dcab4012b532710867d4316e7c26090b495fd3a17077415501ee6"}
Oct 30 13:45:23.444676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.444655    4993 scope.go:122] "RemoveContainer" containerID="cb111761756dcab4012b532710867d4316e7c26090b495fd3a17077415501ee6"
Oct 30 13:45:23.445001 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.444975    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb_odh-ai-gateway-infra(a7bc62a2-0765-497f-8529-8a11acfeb6be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be"
Oct 30 13:45:23.447529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.447502    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" event={"ID":"552e7cb2-282d-4f7e-b4ca-952b9df7a041","Type":"ContainerStarted","Data":"82878e6eb276aea07f17616f9516fe7e458f7b357f0e88cd6729276789e2656f"}
Oct 30 13:45:23.450652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.450626    4993 generic.go:356] "Generic (PLEG): container finished" podID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerID="e6efb607c326f618e638ec514ece569de2466365920978aa141a9a945dd52f22" exitCode=60
Oct 30 13:45:23.450772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.450756    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" event={"ID":"816d95a5-15ed-418c-93f1-79b39600a5e3","Type":"ContainerDied","Data":"e6efb607c326f618e638ec514ece569de2466365920978aa141a9a945dd52f22"}
Oct 30 13:45:23.451154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.451136    4993 scope.go:122] "RemoveContainer" containerID="e6efb607c326f618e638ec514ece569de2466365920978aa141a9a945dd52f22"
Oct 30 13:45:23.451495 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.451462    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh_odh-ai-gateway-infra(816d95a5-15ed-418c-93f1-79b39600a5e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3"
Oct 30 13:45:23.459512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.453952    4993 generic.go:356] "Generic (PLEG): container finished" podID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerID="e2230eaa027005a14fc3294778abf4ed2cf2e5d535b965cee6f2ccc377c02877" exitCode=60
Oct 30 13:45:23.459512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.454029    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" event={"ID":"d8394c36-ec95-44a0-a6d3-a01716310ebc","Type":"ContainerDied","Data":"e2230eaa027005a14fc3294778abf4ed2cf2e5d535b965cee6f2ccc377c02877"}
Oct 30 13:45:23.463818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.463795    4993 scope.go:122] "RemoveContainer" containerID="e2230eaa027005a14fc3294778abf4ed2cf2e5d535b965cee6f2ccc377c02877"
Oct 30 13:45:23.464483 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.464446    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2_odh-ai-gateway-infra(d8394c36-ec95-44a0-a6d3-a01716310ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc"
Oct 30 13:45:23.465732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.465712    4993 generic.go:356] "Generic (PLEG): container finished" podID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerID="e92b702d2a162a07c8175e1c0ecf77871dd9bbadba69828c3a18ea4a50fa33e3" exitCode=60
Oct 30 13:45:23.466112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.465855    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" event={"ID":"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0","Type":"ContainerDied","Data":"e92b702d2a162a07c8175e1c0ecf77871dd9bbadba69828c3a18ea4a50fa33e3"}
Oct 30 13:45:23.466709 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.466692    4993 scope.go:122] "RemoveContainer" containerID="e92b702d2a162a07c8175e1c0ecf77871dd9bbadba69828c3a18ea4a50fa33e3"
Oct 30 13:45:23.467220 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.467184    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb_odh-ai-gateway-infra(a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0"
Oct 30 13:45:23.474783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.474756    4993 generic.go:356] "Generic (PLEG): container finished" podID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerID="937f5d1504abbeec4e909286182201ed9528c02c0c02b3f9f9703cbb4fb33f57" exitCode=60
Oct 30 13:45:23.474935 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.474916    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27" event={"ID":"97c724a9-ac88-4e9e-9d50-c8bf38d366a8","Type":"ContainerDied","Data":"937f5d1504abbeec4e909286182201ed9528c02c0c02b3f9f9703cbb4fb33f57"}
Oct 30 13:45:23.475387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.475367    4993 scope.go:122] "RemoveContainer" containerID="937f5d1504abbeec4e909286182201ed9528c02c0c02b3f9f9703cbb4fb33f57"
Oct 30 13:45:23.476351 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.475707    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27_odh-ai-gateway-infra(97c724a9-ac88-4e9e-9d50-c8bf38d366a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8"
Oct 30 13:45:23.479251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.479032    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" event={"ID":"7f0393d0-1fa2-4248-9d19-f31b08556afa","Type":"ContainerStarted","Data":"e203c29fc83bb6f9fd28bf1cf778b154470587e1d492992656ee3ec4df488ae4"}
Oct 30 13:45:23.486255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.482234    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" event={"ID":"2907709a-ad1d-4724-ab42-02ad7defb4b1","Type":"ContainerStarted","Data":"80a6c15f6f020655eb578f0d5876056b606b3e914d46aa8de8175409b0414e05"}
Oct 30 13:45:23.486255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.484568    4993 generic.go:356] "Generic (PLEG): container finished" podID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerID="e7aa62bd5ced0c266c31e76d19500246f2275a748dfefd35dfc07bd9a8eb4055" exitCode=60
Oct 30 13:45:23.486255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.484636    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" event={"ID":"380b156b-fe53-460f-aa79-2a9f98e37bd2","Type":"ContainerDied","Data":"e7aa62bd5ced0c266c31e76d19500246f2275a748dfefd35dfc07bd9a8eb4055"}
Oct 30 13:45:23.486255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.484979    4993 scope.go:122] "RemoveContainer" containerID="e7aa62bd5ced0c266c31e76d19500246f2275a748dfefd35dfc07bd9a8eb4055"
Oct 30 13:45:23.486255 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.485276    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm_odh-ai-gateway-infra(380b156b-fe53-460f-aa79-2a9f98e37bd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2"
Oct 30 13:45:23.487325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.486859    4993 generic.go:356] "Generic (PLEG): container finished" podID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerID="3a2c462b2a4e70232d3a1c08431eafb408ce027475e656271aa60ae35777b6c7" exitCode=60
Oct 30 13:45:23.487325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.486903    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" event={"ID":"de1bfff9-9805-4b5a-a959-734a9a1570e8","Type":"ContainerDied","Data":"3a2c462b2a4e70232d3a1c08431eafb408ce027475e656271aa60ae35777b6c7"}
Oct 30 13:45:23.487325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.487279    4993 scope.go:122] "RemoveContainer" containerID="3a2c462b2a4e70232d3a1c08431eafb408ce027475e656271aa60ae35777b6c7"
Oct 30 13:45:23.487626 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.487604    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm_odh-ai-gateway-infra(de1bfff9-9805-4b5a-a959-734a9a1570e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8"
Oct 30 13:45:23.489155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.489101    4993 generic.go:356] "Generic (PLEG): container finished" podID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerID="316fdbe5d10c6fff8abb99172b47e73d8ecf0e347f01fa81703e9ff1be2956df" exitCode=60
Oct 30 13:45:23.489155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.489145    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" event={"ID":"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2","Type":"ContainerDied","Data":"316fdbe5d10c6fff8abb99172b47e73d8ecf0e347f01fa81703e9ff1be2956df"}
Oct 30 13:45:23.545873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.545848    4993 scope.go:122] "RemoveContainer" containerID="0ebe92f8ca0985c2da7da6f7671ff023bbbb9794524d8b0513a5cc15dd510221"
Oct 30 13:45:23.579268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.579179    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" podStartSLOduration=23.579159328 podStartE2EDuration="23.579159328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:23.566383762 +0000 UTC m=+4388.042103371" watchObservedRunningTime="2026-10-30 13:45:23.579159328 +0000 UTC m=+4388.054878939"
Oct 30 13:45:23.629235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.628549    4993 scope.go:122] "RemoveContainer" containerID="3eaa7c05312493274a1734abe92bc088e7328d15bc3c07734d3ea7543ed3f9e3"
Oct 30 13:45:23.639278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.639243    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:45:23.689551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.688679    4993 scope.go:122] "RemoveContainer" containerID="bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e"
Oct 30 13:45:23.714012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.713982    4993 scope.go:122] "RemoveContainer" containerID="bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e"
Oct 30 13:45:23.714419 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:23.714389    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e\": container with ID starting with bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e not found: ID does not exist" containerID="bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e"
Oct 30 13:45:23.714524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.714434    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e"} err="failed to get container status \"bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e\": rpc error: code = NotFound desc = could not find container \"bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e\": container with ID starting with bc7ac428490ad6f37b035569ffa21b188e8c9ae5e04dc17c5f8f9f0201de9d9e not found: ID does not exist"
Oct 30 13:45:23.714524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.714454    4993 scope.go:122] "RemoveContainer" containerID="a1e030ce85cf57219640310dff9dc580c5e18d1ae2e1a1defc9de244ebe9dda1"
Oct 30 13:45:23.717492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.717469    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:23.734321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.734296    4993 scope.go:122] "RemoveContainer" containerID="94e714e37b94a4d1afd7e20af0efe6369d099001e46b1b395764e9e1c6b590b2"
Oct 30 13:45:23.758253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.756390    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:23.765455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.765430    4993 scope.go:122] "RemoveContainer" containerID="b46b3bc85e0e5e334033f07d21e3214049ea617f9010d62d3642fd53fc2f7a95"
Oct 30 13:45:23.765836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.765808    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96952ed-4208-4568-9005-6a53e3f36eb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96952ed-4208-4568-9005-6a53e3f36eb6-maas-api-service-ca\") pod \"b96952ed-4208-4568-9005-6a53e3f36eb6\" (UID: \"b96952ed-4208-4568-9005-6a53e3f36eb6\") "
Oct 30 13:45:23.765910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.765897    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-model-cache\") pod \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") "
Oct 30 13:45:23.765995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.765983    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-dshm\") pod \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") "
Oct 30 13:45:23.766049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.766037    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-kube-api-access-d4z8t\" (UniqueName: \"kubernetes.io/projected/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-kube-api-access-d4z8t\") pod \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") "
Oct 30 13:45:23.766136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.766125    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tls-certs\") pod \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") "
Oct 30 13:45:23.766187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.766176    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-home\") pod \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") "
Oct 30 13:45:23.766283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.766265    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tmp-dir\") pod \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\" (UID: \"fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b\") "
Oct 30 13:45:23.766388 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.766363    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96952ed-4208-4568-9005-6a53e3f36eb6-kube-api-access-v8pd5\" (UniqueName: \"kubernetes.io/projected/b96952ed-4208-4568-9005-6a53e3f36eb6-kube-api-access-v8pd5\") pod \"b96952ed-4208-4568-9005-6a53e3f36eb6\" (UID: \"b96952ed-4208-4568-9005-6a53e3f36eb6\") "
Oct 30 13:45:23.766667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.766641    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-model-cache" pod "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" (UID: "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:23.767322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.767023    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tmp-dir" pod "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" (UID: "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:23.767435 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.767075    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-home" pod "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" (UID: "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:23.768054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.767844    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.768054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.767872    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.768054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.767887    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.768355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.768190    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96952ed-4208-4568-9005-6a53e3f36eb6-maas-api-service-ca" pod "b96952ed-4208-4568-9005-6a53e3f36eb6" (UID: "b96952ed-4208-4568-9005-6a53e3f36eb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:23.770727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.770378    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-dshm" pod "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" (UID: "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:23.770727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.770423    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96952ed-4208-4568-9005-6a53e3f36eb6-kube-api-access-v8pd5" pod "b96952ed-4208-4568-9005-6a53e3f36eb6" (UID: "b96952ed-4208-4568-9005-6a53e3f36eb6"). InnerVolumeSpecName "kube-api-access-v8pd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.770903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.770731    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-kube-api-access-d4z8t" pod "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" (UID: "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b"). InnerVolumeSpecName "kube-api-access-d4z8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.773412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.773381    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tls-certs" pod "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" (UID: "fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:23.846069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.846044    4993 scope.go:122] "RemoveContainer" containerID="aa02fab57371e233fff768d9f8fdafcf6e39139830edcf74738d8b7539591253"
Oct 30 13:45:23.866268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.866240    4993 scope.go:122] "RemoveContainer" containerID="a102c923dfa34b61c5fe96732c04b51e6c4ea2762f0f731f76fe5c8657ec870e"
Oct 30 13:45:23.868921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.868885    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-maas-api-service-ca\") pod \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\" (UID: \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\") "
Oct 30 13:45:23.869052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.869014    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-kube-api-access-cgdk5\" (UniqueName: \"kubernetes.io/projected/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-kube-api-access-cgdk5\") pod \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\" (UID: \"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2\") "
Oct 30 13:45:23.869229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.869189    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-maas-api-service-ca" pod "3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" (UID: "3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:23.869838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.869817    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.869938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.869841    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8pd5\" (UniqueName: \"kubernetes.io/projected/b96952ed-4208-4568-9005-6a53e3f36eb6-kube-api-access-v8pd5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.869938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.869852    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96952ed-4208-4568-9005-6a53e3f36eb6-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.869938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.869862    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.869938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.869873    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4z8t\" (UniqueName: \"kubernetes.io/projected/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b-kube-api-access-d4z8t\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.869938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.869882    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.871587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.871556    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-kube-api-access-cgdk5" pod "3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" (UID: "3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2"). InnerVolumeSpecName "kube-api-access-cgdk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.972310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:23.972276    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdk5\" (UniqueName: \"kubernetes.io/projected/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2-kube-api-access-cgdk5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:24.011522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.011158    4993 scope.go:122] "RemoveContainer" containerID="a97a40cd0d53f263564d082e6216291d65151270af5ea8640114cc2ab928c8e5"
Oct 30 13:45:24.111860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.111836    4993 scope.go:122] "RemoveContainer" containerID="b5b00b578b98f5f666a082a67f37ea1b3353cb22a89843d039f187ab0448a11f"
Oct 30 13:45:24.129473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.129441    4993 scope.go:122] "RemoveContainer" containerID="5b817eba7ecac6613ced24bcb35dd54101c82d26bacfde1a5e69ebb419612331"
Oct 30 13:45:24.151033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.151001    4993 scope.go:122] "RemoveContainer" containerID="3345d718bb7bd8aec7d0a9ec5be1d09d98e5130e9559fd900342539c04505a24"
Oct 30 13:45:24.221812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.217397    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"]
Oct 30 13:45:24.228943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.228907    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-09ad1e/test-model-09ad1e-kserve-658d8b4df9-kn6pk"]
Oct 30 13:45:24.306610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.306571    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"]
Oct 30 13:45:24.314664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.314629    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-025c7f-29889465-xw782"]
Oct 30 13:45:24.333432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.333406    4993 scope.go:122] "RemoveContainer" containerID="18255b0eb7c8cb0653a1e79fbf321f055d2cfdfd79a904b9c3a62b76213a5035"
Oct 30 13:45:24.364140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.364096    4993 scope.go:122] "RemoveContainer" containerID="1580b838891b13f3aef1d83e4b4112aa1424a6e9c58436d602c5813c861a7887"
Oct 30 13:45:24.383163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.382916    4993 scope.go:122] "RemoveContainer" containerID="f312a3e4e55684ca0b5d3cb1fe1e38d5893e49feffde32fffc628410118f60b9"
Oct 30 13:45:24.394366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.394326    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65" podUID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:24.515061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.515026    4993 generic.go:356] "Generic (PLEG): container finished" podID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerID="82878e6eb276aea07f17616f9516fe7e458f7b357f0e88cd6729276789e2656f" exitCode=60
Oct 30 13:45:24.515446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.515100    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" event={"ID":"552e7cb2-282d-4f7e-b4ca-952b9df7a041","Type":"ContainerDied","Data":"82878e6eb276aea07f17616f9516fe7e458f7b357f0e88cd6729276789e2656f"}
Oct 30 13:45:24.515446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.515219    4993 scope.go:122] "RemoveContainer" containerID="a607b50c5d6427b9b38c6f660674d9cdf682513f46cfef94440825753ecd7cfe"
Oct 30 13:45:24.515705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.515688    4993 scope.go:122] "RemoveContainer" containerID="82878e6eb276aea07f17616f9516fe7e458f7b357f0e88cd6729276789e2656f"
Oct 30 13:45:24.516196 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:24.516153    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg_odh-ai-gateway-infra(552e7cb2-282d-4f7e-b4ca-952b9df7a041)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041"
Oct 30 13:45:24.519115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.519086    4993 generic.go:356] "Generic (PLEG): container finished" podID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerID="22aee46f66987a6bdd0e767455db5f68e1af24471fc2956658de417d12d118f3" exitCode=60
Oct 30 13:45:24.519790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.519160    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" event={"ID":"9325a541-f5aa-4272-bbe1-c75fd95e384a","Type":"ContainerDied","Data":"22aee46f66987a6bdd0e767455db5f68e1af24471fc2956658de417d12d118f3"}
Oct 30 13:45:24.519790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.519694    4993 scope.go:122] "RemoveContainer" containerID="22aee46f66987a6bdd0e767455db5f68e1af24471fc2956658de417d12d118f3"
Oct 30 13:45:24.520129 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:24.520101    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8_odh-ai-gateway-infra(9325a541-f5aa-4272-bbe1-c75fd95e384a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a"
Oct 30 13:45:24.534099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.534064    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"]
Oct 30 13:45:24.536428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.536401    4993 generic.go:356] "Generic (PLEG): container finished" podID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerID="f53c0970f587bbcce7d4d16b5630687c279ee9c5f7afe9cac34e375c6cd32760" exitCode=60
Oct 30 13:45:24.536629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.536554    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" event={"ID":"7febc0c4-d800-4a15-b9b8-708d4517a176","Type":"ContainerDied","Data":"f53c0970f587bbcce7d4d16b5630687c279ee9c5f7afe9cac34e375c6cd32760"}
Oct 30 13:45:24.537102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.537082    4993 scope.go:122] "RemoveContainer" containerID="f53c0970f587bbcce7d4d16b5630687c279ee9c5f7afe9cac34e375c6cd32760"
Oct 30 13:45:24.537385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.537359    4993 scope.go:122] "RemoveContainer" containerID="f8835816482613c783d7fae8974012eb1c8e002b63f46b16cccc5a17ba0bad27"
Oct 30 13:45:24.537543 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:24.537523    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz_odh-ai-gateway-infra(7febc0c4-d800-4a15-b9b8-708d4517a176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176"
Oct 30 13:45:24.539785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.539761    4993 generic.go:356] "Generic (PLEG): container finished" podID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerID="e203c29fc83bb6f9fd28bf1cf778b154470587e1d492992656ee3ec4df488ae4" exitCode=60
Oct 30 13:45:24.539869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.539853    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" event={"ID":"7f0393d0-1fa2-4248-9d19-f31b08556afa","Type":"ContainerDied","Data":"e203c29fc83bb6f9fd28bf1cf778b154470587e1d492992656ee3ec4df488ae4"}
Oct 30 13:45:24.540349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.540328    4993 scope.go:122] "RemoveContainer" containerID="e203c29fc83bb6f9fd28bf1cf778b154470587e1d492992656ee3ec4df488ae4"
Oct 30 13:45:24.540767 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:24.540740    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8_odh-ai-gateway-infra(7f0393d0-1fa2-4248-9d19-f31b08556afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa"
Oct 30 13:45:24.542718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.542693    4993 generic.go:356] "Generic (PLEG): container finished" podID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerID="80a6c15f6f020655eb578f0d5876056b606b3e914d46aa8de8175409b0414e05" exitCode=60
Oct 30 13:45:24.542805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.542763    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" event={"ID":"2907709a-ad1d-4724-ab42-02ad7defb4b1","Type":"ContainerDied","Data":"80a6c15f6f020655eb578f0d5876056b606b3e914d46aa8de8175409b0414e05"}
Oct 30 13:45:24.543445 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.543421    4993 scope.go:122] "RemoveContainer" containerID="80a6c15f6f020655eb578f0d5876056b606b3e914d46aa8de8175409b0414e05"
Oct 30 13:45:24.543957 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:24.543927    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8_odh-ai-gateway-infra(2907709a-ad1d-4724-ab42-02ad7defb4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1"
Oct 30 13:45:24.609939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.609894    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"
Oct 30 13:45:24.610953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.609897    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh" event={"ID":"3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2","Type":"ContainerDied","Data":"60b34d1f556b00c9840e4aebb75b19dc72092864e2fc71df6902cf64134021b1"}
Oct 30 13:45:24.626349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.626321    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"]
Oct 30 13:45:24.627534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.626427    4993 generic.go:356] "Generic (PLEG): container finished" podID="49589327-9884-4386-a522-df5b12d23094" containerID="4e8cec9321f333e7ca11beb78101888190228d2be6d35144025e473f67d1f97a" exitCode=60
Oct 30 13:45:24.630477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.626952    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" event={"ID":"49589327-9884-4386-a522-df5b12d23094","Type":"ContainerDied","Data":"4e8cec9321f333e7ca11beb78101888190228d2be6d35144025e473f67d1f97a"}
Oct 30 13:45:24.630477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.630454    4993 scope.go:122] "RemoveContainer" containerID="9f2673c69992922314806e9f21edd9c2ffe5fc1d933338763f0607ebb90a3c2c"
Oct 30 13:45:24.633023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.630883    4993 scope.go:122] "RemoveContainer" containerID="4e8cec9321f333e7ca11beb78101888190228d2be6d35144025e473f67d1f97a"
Oct 30 13:45:24.633320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.630975    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8" event={"ID":"b96952ed-4208-4568-9005-6a53e3f36eb6","Type":"ContainerDied","Data":"70c90cb30fd2914ae7fb7194db0a2f0217ff9cdd5b83ad2e92d1fdbf1f31d818"}
Oct 30 13:45:24.633407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.631028    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"
Oct 30 13:45:24.633812 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:24.633786    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg_odh-ai-gateway-infra(49589327-9884-4386-a522-df5b12d23094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" podUID="49589327-9884-4386-a522-df5b12d23094"
Oct 30 13:45:24.638322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.638299    4993 generic.go:356] "Generic (PLEG): container finished" podID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerID="f050d6f3c203a8c04522da8ee82cdedffce701fd7d68150404e5edc8470da2e0" exitCode=60
Oct 30 13:45:24.638453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.638426    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" event={"ID":"998580a2-a1bf-4eb6-b8db-b3277af46957","Type":"ContainerDied","Data":"f050d6f3c203a8c04522da8ee82cdedffce701fd7d68150404e5edc8470da2e0"}
Oct 30 13:45:24.638939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.638911    4993 scope.go:122] "RemoveContainer" containerID="f050d6f3c203a8c04522da8ee82cdedffce701fd7d68150404e5edc8470da2e0"
Oct 30 13:45:24.639617 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:24.639312    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9_odh-ai-gateway-infra(998580a2-a1bf-4eb6-b8db-b3277af46957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957"
Oct 30 13:45:24.644737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.644433    4993 generic.go:356] "Generic (PLEG): container finished" podID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerID="f63269caea7adbef39ee6b58d02c56f3fddaf11cc1ba84cdeb3718bc46fa2fa3" exitCode=60
Oct 30 13:45:24.644737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.644545    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" event={"ID":"59ac7f13-0f86-43e3-9189-d89aa39defb9","Type":"ContainerDied","Data":"f63269caea7adbef39ee6b58d02c56f3fddaf11cc1ba84cdeb3718bc46fa2fa3"}
Oct 30 13:45:24.646690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.646474    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"]
Oct 30 13:45:24.669247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.651242    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"
Oct 30 13:45:24.847057 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:24.845438    4993 scope.go:122] "RemoveContainer" containerID="d86bab65fba7dd522607859003bced0e3894cc12c424b887a1ff2d416759990e"
Oct 30 13:45:25.008829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.008504    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"]
Oct 30 13:45:25.050708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.050672    4993 scope.go:122] "RemoveContainer" containerID="1748aaf25bb7978775531c0b742434109882d2457e718cfb933791a52102d0a9"
Oct 30 13:45:25.052550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.052305    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"]
Oct 30 13:45:25.083975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.083589    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"]
Oct 30 13:45:25.119519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.105481    4993 scope.go:122] "RemoveContainer" containerID="316fdbe5d10c6fff8abb99172b47e73d8ecf0e347f01fa81703e9ff1be2956df"
Oct 30 13:45:25.125439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.125193    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"]
Oct 30 13:45:25.203418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.197414    4993 scope.go:122] "RemoveContainer" containerID="093d43232dde3539ecb5a71e0daa7382f8ca1db68788b212819795224c73c74f"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.506087    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"]
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507308    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507335    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507360    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5bf8bf0b-166c-44c4-be3e-0b8488950bb3" containerName="main"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507367    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf8bf0b-166c-44c4-be3e-0b8488950bb3" containerName="main"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507412    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507420    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507441    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507450    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507468    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507474    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507496    4993 cpu_manager.go:414] "containerMap: removing container" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507502    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507531    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" containerName="main"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507537    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" containerName="main"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507557    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507563    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507575    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507581    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507596    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507602    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507621    4993 cpu_manager.go:414] "containerMap: removing container" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507629    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507650    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507656    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507669    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507675    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507863    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" containerName="main"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507877    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507886    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507894    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507904    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507913    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507924    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507934    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507942    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf8bf0b-166c-44c4-be3e-0b8488950bb3" containerName="main"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507953    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:25.522640 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.507964    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a48c09-e2e1-4006-8d0c-2331997f7a85" containerName="cleanup"
Oct 30 13:45:25.604156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.568408    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.717987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.647301    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:45:25.717987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.700009    4993 scope.go:122] "RemoveContainer" containerID="947e0a22542b20d0accb64c0e849508dd920efff7325a0ed033802b17cab046c"
Oct 30 13:45:25.741315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.741163    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.779014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.748652    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29jn\" (UniqueName: \"kubernetes.io/projected/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-kube-api-access-l29jn\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.779014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.748730    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.835517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.834630    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf8bf0b-166c-44c4-be3e-0b8488950bb3" path="/var/lib/kubelet/pods/5bf8bf0b-166c-44c4-be3e-0b8488950bb3/volumes"
Oct 30 13:45:25.835517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.835402    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" path="/var/lib/kubelet/pods/8e7f69c1-5545-4e9c-8aee-2046fb00d5c6/volumes"
Oct 30 13:45:25.835998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.835975    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"]
Oct 30 13:45:25.837270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.837250    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"]
Oct 30 13:45:25.837443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.837427    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"]
Oct 30 13:45:25.842531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.842494    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"]
Oct 30 13:45:25.850537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.850463    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.850748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.850677    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29jn\" (UniqueName: \"kubernetes.io/projected/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-kube-api-access-l29jn\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.850748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.850726    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.852363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.851918    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.867550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.867485    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:25.892569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.891430    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"]
Oct 30 13:45:25.937405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.935851    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"]
Oct 30 13:45:25.982524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.982491    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:25.998139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:25.998101    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"]
Oct 30 13:45:26.061789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.060904    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29jn\" (UniqueName: \"kubernetes.io/projected/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-kube-api-access-l29jn\") pod \"maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:26.072132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.068032    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ac7f13-0f86-43e3-9189-d89aa39defb9-kube-api-access-qkzjs\" (UniqueName: \"kubernetes.io/projected/59ac7f13-0f86-43e3-9189-d89aa39defb9-kube-api-access-qkzjs\") pod \"59ac7f13-0f86-43e3-9189-d89aa39defb9\" (UID: \"59ac7f13-0f86-43e3-9189-d89aa39defb9\") "
Oct 30 13:45:26.072132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.068402    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ac7f13-0f86-43e3-9189-d89aa39defb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ac7f13-0f86-43e3-9189-d89aa39defb9-maas-api-service-ca\") pod \"59ac7f13-0f86-43e3-9189-d89aa39defb9\" (UID: \"59ac7f13-0f86-43e3-9189-d89aa39defb9\") "
Oct 30 13:45:26.099327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.098512    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"]
Oct 30 13:45:26.107301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.103819    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ac7f13-0f86-43e3-9189-d89aa39defb9-maas-api-service-ca" pod "59ac7f13-0f86-43e3-9189-d89aa39defb9" (UID: "59ac7f13-0f86-43e3-9189-d89aa39defb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.122118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.117986    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ac7f13-0f86-43e3-9189-d89aa39defb9-kube-api-access-qkzjs" pod "59ac7f13-0f86-43e3-9189-d89aa39defb9" (UID: "59ac7f13-0f86-43e3-9189-d89aa39defb9"). InnerVolumeSpecName "kube-api-access-qkzjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.127114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.126697    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"]
Oct 30 13:45:26.129905 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.129699    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy" containerID="cri-o://3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94" gracePeriod=30
Oct 30 13:45:26.169083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.169039    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:26.211848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.202285    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzjs\" (UniqueName: \"kubernetes.io/projected/59ac7f13-0f86-43e3-9189-d89aa39defb9-kube-api-access-qkzjs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.211848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.202358    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ac7f13-0f86-43e3-9189-d89aa39defb9-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.211848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.202399    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"]
Oct 30 13:45:26.211848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.204282    4993 scope.go:122] "RemoveContainer" containerID="835117598a783b80b0f477b7316de8825b438e2e5117cd1487b3f585bcc4a3e9"
Oct 30 13:45:26.232569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.231474    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a93bd-29889465-mdpj8"]
Oct 30 13:45:26.232569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.231721    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:26.257486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.257448    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"]
Oct 30 13:45:26.271760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.271684    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"]
Oct 30 13:45:26.276919 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.276846    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f49b40-29889465-h4cbh"]
Oct 30 13:45:26.297812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.297769    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:26.310394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.308861    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613f276e-cad4-40e0-84e8-b5c7e2524853-kube-api-access-mbpn7\" (UniqueName: \"kubernetes.io/projected/613f276e-cad4-40e0-84e8-b5c7e2524853-kube-api-access-mbpn7\") pod \"613f276e-cad4-40e0-84e8-b5c7e2524853\" (UID: \"613f276e-cad4-40e0-84e8-b5c7e2524853\") "
Oct 30 13:45:26.310394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.308963    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613f276e-cad4-40e0-84e8-b5c7e2524853-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613f276e-cad4-40e0-84e8-b5c7e2524853-maas-api-service-ca\") pod \"613f276e-cad4-40e0-84e8-b5c7e2524853\" (UID: \"613f276e-cad4-40e0-84e8-b5c7e2524853\") "
Oct 30 13:45:26.312136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.312099    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613f276e-cad4-40e0-84e8-b5c7e2524853-maas-api-service-ca" pod "613f276e-cad4-40e0-84e8-b5c7e2524853" (UID: "613f276e-cad4-40e0-84e8-b5c7e2524853"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.343220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.343149    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:26.361250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.360744    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"]
Oct 30 13:45:26.366311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.365864    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613f276e-cad4-40e0-84e8-b5c7e2524853-kube-api-access-mbpn7" pod "613f276e-cad4-40e0-84e8-b5c7e2524853" (UID: "613f276e-cad4-40e0-84e8-b5c7e2524853"). InnerVolumeSpecName "kube-api-access-mbpn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.377602 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.376656    4993 scope.go:122] "RemoveContainer" containerID="3a59798e0578c8bc91ef567f653b0d1201a3027734377aacf80ca36ebe2e4806"
Oct 30 13:45:26.414471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.396226    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"]
Oct 30 13:45:26.427425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.426031    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75087b72-ac15-4d81-bd96-659ac0f7b2ef-kube-api-access-q54nt\" (UniqueName: \"kubernetes.io/projected/75087b72-ac15-4d81-bd96-659ac0f7b2ef-kube-api-access-q54nt\") pod \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\" (UID: \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\") "
Oct 30 13:45:26.427425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.426177    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75087b72-ac15-4d81-bd96-659ac0f7b2ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75087b72-ac15-4d81-bd96-659ac0f7b2ef-maas-api-service-ca\") pod \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\" (UID: \"75087b72-ac15-4d81-bd96-659ac0f7b2ef\") "
Oct 30 13:45:26.427425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.426911    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbpn7\" (UniqueName: \"kubernetes.io/projected/613f276e-cad4-40e0-84e8-b5c7e2524853-kube-api-access-mbpn7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.427425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.426935    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613f276e-cad4-40e0-84e8-b5c7e2524853-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.427425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.427352    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75087b72-ac15-4d81-bd96-659ac0f7b2ef-maas-api-service-ca" pod "75087b72-ac15-4d81-bd96-659ac0f7b2ef" (UID: "75087b72-ac15-4d81-bd96-659ac0f7b2ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.439529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.439443    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:26.466509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.466305    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:26.475256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.475168    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"]
Oct 30 13:45:26.530123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.528384    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-kube-api-access-mmkw6\" (UniqueName: \"kubernetes.io/projected/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-kube-api-access-mmkw6\") pod \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\" (UID: \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\") "
Oct 30 13:45:26.530123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.528440    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-maas-api-service-ca\") pod \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\" (UID: \"ebf48e8e-316b-4250-9bae-9bcb736d3cb9\") "
Oct 30 13:45:26.530123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.529285    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75087b72-ac15-4d81-bd96-659ac0f7b2ef-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.530123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.529662    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-maas-api-service-ca" pod "ebf48e8e-316b-4250-9bae-9bcb736d3cb9" (UID: "ebf48e8e-316b-4250-9bae-9bcb736d3cb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.542802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.542709    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:26.544271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.543827    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"]
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544858    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544880    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544893    4993 cpu_manager.go:414] "containerMap: removing container" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544898    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544909    4993 cpu_manager.go:414] "containerMap: removing container" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544913    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544921    4993 cpu_manager.go:414] "containerMap: removing container" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544926    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544939    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544944    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544954    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544962    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544987    4993 cpu_manager.go:414] "containerMap: removing container" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.544995    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545010    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545016    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545034    4993 cpu_manager.go:414] "containerMap: removing container" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545042    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545057    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545062    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545069    4993 cpu_manager.go:414] "containerMap: removing container" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545074    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545084    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545088    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545102    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545109    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545119    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545123    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545135    4993 cpu_manager.go:414] "containerMap: removing container" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545139    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545149    4993 cpu_manager.go:414] "containerMap: removing container" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:26.545255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545154    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545833    4993 cpu_manager.go:414] "containerMap: removing container" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545892    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545918    4993 cpu_manager.go:414] "containerMap: removing container" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.545926    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546173    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546191    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546201    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546228    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546239    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546250    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546272    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546287    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546295    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546306    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546322    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546330    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546348    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546360    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7f69c1-5545-4e9c-8aee-2046fb00d5c6" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546370    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546378    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" containerName="cleanup"
Oct 30 13:45:26.547695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.546390    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:26.550086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.550048    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75087b72-ac15-4d81-bd96-659ac0f7b2ef-kube-api-access-q54nt" pod "75087b72-ac15-4d81-bd96-659ac0f7b2ef" (UID: "75087b72-ac15-4d81-bd96-659ac0f7b2ef"). InnerVolumeSpecName "kube-api-access-q54nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.551880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.551262    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.582259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.578470    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-kube-api-access-mmkw6" pod "ebf48e8e-316b-4250-9bae-9bcb736d3cb9" (UID: "ebf48e8e-316b-4250-9bae-9bcb736d3cb9"). InnerVolumeSpecName "kube-api-access-mmkw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.619343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.618612    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:26.619477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.619441    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"]
Oct 30 13:45:26.623877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.622970    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:26.630695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.630608    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66456109-7df3-4de7-978e-13496801fff2-kube-api-access-xn2wp\" (UniqueName: \"kubernetes.io/projected/66456109-7df3-4de7-978e-13496801fff2-kube-api-access-xn2wp\") pod \"66456109-7df3-4de7-978e-13496801fff2\" (UID: \"66456109-7df3-4de7-978e-13496801fff2\") "
Oct 30 13:45:26.631297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.630923    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7bc62a2-0765-497f-8529-8a11acfeb6be-kube-api-access-lpv4f\" (UniqueName: \"kubernetes.io/projected/a7bc62a2-0765-497f-8529-8a11acfeb6be-kube-api-access-lpv4f\") pod \"a7bc62a2-0765-497f-8529-8a11acfeb6be\" (UID: \"a7bc62a2-0765-497f-8529-8a11acfeb6be\") "
Oct 30 13:45:26.631297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.630982    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66456109-7df3-4de7-978e-13496801fff2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66456109-7df3-4de7-978e-13496801fff2-maas-api-service-ca\") pod \"66456109-7df3-4de7-978e-13496801fff2\" (UID: \"66456109-7df3-4de7-978e-13496801fff2\") "
Oct 30 13:45:26.631297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.631096    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7bc62a2-0765-497f-8529-8a11acfeb6be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bc62a2-0765-497f-8529-8a11acfeb6be-maas-api-service-ca\") pod \"a7bc62a2-0765-497f-8529-8a11acfeb6be\" (UID: \"a7bc62a2-0765-497f-8529-8a11acfeb6be\") "
Oct 30 13:45:26.632279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.632255    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q54nt\" (UniqueName: \"kubernetes.io/projected/75087b72-ac15-4d81-bd96-659ac0f7b2ef-kube-api-access-q54nt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.632413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.632399    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmkw6\" (UniqueName: \"kubernetes.io/projected/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-kube-api-access-mmkw6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.632512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.632501    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf48e8e-316b-4250-9bae-9bcb736d3cb9-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.632924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.632903    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bc62a2-0765-497f-8529-8a11acfeb6be-maas-api-service-ca" pod "a7bc62a2-0765-497f-8529-8a11acfeb6be" (UID: "a7bc62a2-0765-497f-8529-8a11acfeb6be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.633469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.633449    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66456109-7df3-4de7-978e-13496801fff2-maas-api-service-ca" pod "66456109-7df3-4de7-978e-13496801fff2" (UID: "66456109-7df3-4de7-978e-13496801fff2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.637860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.637834    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:26.641638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.641611    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66456109-7df3-4de7-978e-13496801fff2-kube-api-access-xn2wp" pod "66456109-7df3-4de7-978e-13496801fff2" (UID: "66456109-7df3-4de7-978e-13496801fff2"). InnerVolumeSpecName "kube-api-access-xn2wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.644266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.644243    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7bc62a2-0765-497f-8529-8a11acfeb6be-kube-api-access-lpv4f" pod "a7bc62a2-0765-497f-8529-8a11acfeb6be" (UID: "a7bc62a2-0765-497f-8529-8a11acfeb6be"). InnerVolumeSpecName "kube-api-access-lpv4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.649409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.649378    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:26.669860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.668940    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"]
Oct 30 13:45:26.725229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.715298    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"]
Oct 30 13:45:26.725229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.722454    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"]
Oct 30 13:45:26.730908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.730843    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-025c7f/test-model-025c7f-kserve-59878bf66f-fxm65"]
Oct 30 13:45:26.731968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.731930    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:26.734364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.733785    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49c1907-0584-44be-8022-3b27cc76ec35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49c1907-0584-44be-8022-3b27cc76ec35-maas-api-service-ca\") pod \"f49c1907-0584-44be-8022-3b27cc76ec35\" (UID: \"f49c1907-0584-44be-8022-3b27cc76ec35\") "
Oct 30 13:45:26.734364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.733846    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8394c36-ec95-44a0-a6d3-a01716310ebc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8394c36-ec95-44a0-a6d3-a01716310ebc-maas-api-service-ca\") pod \"d8394c36-ec95-44a0-a6d3-a01716310ebc\" (UID: \"d8394c36-ec95-44a0-a6d3-a01716310ebc\") "
Oct 30 13:45:26.735074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.734639    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49c1907-0584-44be-8022-3b27cc76ec35-kube-api-access-m2w2t\" (UniqueName: \"kubernetes.io/projected/f49c1907-0584-44be-8022-3b27cc76ec35-kube-api-access-m2w2t\") pod \"f49c1907-0584-44be-8022-3b27cc76ec35\" (UID: \"f49c1907-0584-44be-8022-3b27cc76ec35\") "
Oct 30 13:45:26.735074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.734931    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8394c36-ec95-44a0-a6d3-a01716310ebc-kube-api-access-tg9wr\" (UniqueName: \"kubernetes.io/projected/d8394c36-ec95-44a0-a6d3-a01716310ebc-kube-api-access-tg9wr\") pod \"d8394c36-ec95-44a0-a6d3-a01716310ebc\" (UID: \"d8394c36-ec95-44a0-a6d3-a01716310ebc\") "
Oct 30 13:45:26.735074 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.734970    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816d95a5-15ed-418c-93f1-79b39600a5e3-kube-api-access-jd449\" (UniqueName: \"kubernetes.io/projected/816d95a5-15ed-418c-93f1-79b39600a5e3-kube-api-access-jd449\") pod \"816d95a5-15ed-418c-93f1-79b39600a5e3\" (UID: \"816d95a5-15ed-418c-93f1-79b39600a5e3\") "
Oct 30 13:45:26.735687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.735612    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816d95a5-15ed-418c-93f1-79b39600a5e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d95a5-15ed-418c-93f1-79b39600a5e3-maas-api-service-ca\") pod \"816d95a5-15ed-418c-93f1-79b39600a5e3\" (UID: \"816d95a5-15ed-418c-93f1-79b39600a5e3\") "
Oct 30 13:45:26.736310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.736231    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.738305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.736668    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66wc\" (UniqueName: \"kubernetes.io/projected/a1faa809-72e9-49a2-8fa4-e97d9de1a166-kube-api-access-c66wc\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.738305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.736770    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.738305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.736961    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66456109-7df3-4de7-978e-13496801fff2-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.738305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.736976    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bc62a2-0765-497f-8529-8a11acfeb6be-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.738305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.736989    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2wp\" (UniqueName: \"kubernetes.io/projected/66456109-7df3-4de7-978e-13496801fff2-kube-api-access-xn2wp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.738305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.737003    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpv4f\" (UniqueName: \"kubernetes.io/projected/a7bc62a2-0765-497f-8529-8a11acfeb6be-kube-api-access-lpv4f\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.739446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.739173    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49c1907-0584-44be-8022-3b27cc76ec35-maas-api-service-ca" pod "f49c1907-0584-44be-8022-3b27cc76ec35" (UID: "f49c1907-0584-44be-8022-3b27cc76ec35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.743590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.739704    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8394c36-ec95-44a0-a6d3-a01716310ebc-maas-api-service-ca" pod "d8394c36-ec95-44a0-a6d3-a01716310ebc" (UID: "d8394c36-ec95-44a0-a6d3-a01716310ebc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.748115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.748072    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816d95a5-15ed-418c-93f1-79b39600a5e3-maas-api-service-ca" pod "816d95a5-15ed-418c-93f1-79b39600a5e3" (UID: "816d95a5-15ed-418c-93f1-79b39600a5e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.805991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.800252    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8394c36-ec95-44a0-a6d3-a01716310ebc-kube-api-access-tg9wr" pod "d8394c36-ec95-44a0-a6d3-a01716310ebc" (UID: "d8394c36-ec95-44a0-a6d3-a01716310ebc"). InnerVolumeSpecName "kube-api-access-tg9wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.809446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.809397    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49c1907-0584-44be-8022-3b27cc76ec35-kube-api-access-m2w2t" pod "f49c1907-0584-44be-8022-3b27cc76ec35" (UID: "f49c1907-0584-44be-8022-3b27cc76ec35"). InnerVolumeSpecName "kube-api-access-m2w2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.810313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.810269    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816d95a5-15ed-418c-93f1-79b39600a5e3-kube-api-access-jd449" pod "816d95a5-15ed-418c-93f1-79b39600a5e3" (UID: "816d95a5-15ed-418c-93f1-79b39600a5e3"). InnerVolumeSpecName "kube-api-access-jd449". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.819599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.819484    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.843272    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998580a2-a1bf-4eb6-b8db-b3277af46957-kube-api-access-r8tq7\" (UniqueName: \"kubernetes.io/projected/998580a2-a1bf-4eb6-b8db-b3277af46957-kube-api-access-r8tq7\") pod \"998580a2-a1bf-4eb6-b8db-b3277af46957\" (UID: \"998580a2-a1bf-4eb6-b8db-b3277af46957\") "
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.843469    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0393d0-1fa2-4248-9d19-f31b08556afa-kube-api-access-4k56z\" (UniqueName: \"kubernetes.io/projected/7f0393d0-1fa2-4248-9d19-f31b08556afa-kube-api-access-4k56z\") pod \"7f0393d0-1fa2-4248-9d19-f31b08556afa\" (UID: \"7f0393d0-1fa2-4248-9d19-f31b08556afa\") "
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.843537    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552e7cb2-282d-4f7e-b4ca-952b9df7a041-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e7cb2-282d-4f7e-b4ca-952b9df7a041-maas-api-service-ca\") pod \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\" (UID: \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\") "
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.843623    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998580a2-a1bf-4eb6-b8db-b3277af46957-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998580a2-a1bf-4eb6-b8db-b3277af46957-maas-api-service-ca\") pod \"998580a2-a1bf-4eb6-b8db-b3277af46957\" (UID: \"998580a2-a1bf-4eb6-b8db-b3277af46957\") "
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.843762    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0393d0-1fa2-4248-9d19-f31b08556afa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0393d0-1fa2-4248-9d19-f31b08556afa-maas-api-service-ca\") pod \"7f0393d0-1fa2-4248-9d19-f31b08556afa\" (UID: \"7f0393d0-1fa2-4248-9d19-f31b08556afa\") "
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.843861    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552e7cb2-282d-4f7e-b4ca-952b9df7a041-kube-api-access-wv696\" (UniqueName: \"kubernetes.io/projected/552e7cb2-282d-4f7e-b4ca-952b9df7a041-kube-api-access-wv696\") pod \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\" (UID: \"552e7cb2-282d-4f7e-b4ca-952b9df7a041\") "
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844114    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844424    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66wc\" (UniqueName: \"kubernetes.io/projected/a1faa809-72e9-49a2-8fa4-e97d9de1a166-kube-api-access-c66wc\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844527    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844719    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d95a5-15ed-418c-93f1-79b39600a5e3-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844739    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49c1907-0584-44be-8022-3b27cc76ec35-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844754    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8394c36-ec95-44a0-a6d3-a01716310ebc-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844771    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2w2t\" (UniqueName: \"kubernetes.io/projected/f49c1907-0584-44be-8022-3b27cc76ec35-kube-api-access-m2w2t\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844787    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg9wr\" (UniqueName: \"kubernetes.io/projected/d8394c36-ec95-44a0-a6d3-a01716310ebc-kube-api-access-tg9wr\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.844801    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd449\" (UniqueName: \"kubernetes.io/projected/816d95a5-15ed-418c-93f1-79b39600a5e3-kube-api-access-jd449\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.845657    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.847988    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0393d0-1fa2-4248-9d19-f31b08556afa-maas-api-service-ca" pod "7f0393d0-1fa2-4248-9d19-f31b08556afa" (UID: "7f0393d0-1fa2-4248-9d19-f31b08556afa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.848436    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552e7cb2-282d-4f7e-b4ca-952b9df7a041-maas-api-service-ca" pod "552e7cb2-282d-4f7e-b4ca-952b9df7a041" (UID: "552e7cb2-282d-4f7e-b4ca-952b9df7a041"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.848880    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998580a2-a1bf-4eb6-b8db-b3277af46957-kube-api-access-r8tq7" pod "998580a2-a1bf-4eb6-b8db-b3277af46957" (UID: "998580a2-a1bf-4eb6-b8db-b3277af46957"). InnerVolumeSpecName "kube-api-access-r8tq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.849864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.849387    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998580a2-a1bf-4eb6-b8db-b3277af46957-maas-api-service-ca" pod "998580a2-a1bf-4eb6-b8db-b3277af46957" (UID: "998580a2-a1bf-4eb6-b8db-b3277af46957"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.855362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.854957    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0393d0-1fa2-4248-9d19-f31b08556afa-kube-api-access-4k56z" pod "7f0393d0-1fa2-4248-9d19-f31b08556afa" (UID: "7f0393d0-1fa2-4248-9d19-f31b08556afa"). InnerVolumeSpecName "kube-api-access-4k56z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.908462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.881171    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.908462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.906278    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552e7cb2-282d-4f7e-b4ca-952b9df7a041-kube-api-access-wv696" pod "552e7cb2-282d-4f7e-b4ca-952b9df7a041" (UID: "552e7cb2-282d-4f7e-b4ca-952b9df7a041"). InnerVolumeSpecName "kube-api-access-wv696". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.908462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.907741    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8" event={"ID":"7f0393d0-1fa2-4248-9d19-f31b08556afa","Type":"ContainerDied","Data":"130b9a7dab2a8e72c179e325b783105ef7cb33182f07f373657863a1363d5c7e"}
Oct 30 13:45:26.908462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.907829    4993 scope.go:122] "RemoveContainer" containerID="e203c29fc83bb6f9fd28bf1cf778b154470587e1d492992656ee3ec4df488ae4"
Oct 30 13:45:26.908462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.907959    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"
Oct 30 13:45:26.908901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.908843    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66wc\" (UniqueName: \"kubernetes.io/projected/a1faa809-72e9-49a2-8fa4-e97d9de1a166-kube-api-access-c66wc\") pod \"maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:26.924829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.923128    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8" event={"ID":"2907709a-ad1d-4724-ab42-02ad7defb4b1","Type":"ContainerDied","Data":"62bb626b3c735df708949bc56610124be044b8dbadd909d895964d52f73ffd1a"}
Oct 30 13:45:26.924829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.923190    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62bb626b3c735df708949bc56610124be044b8dbadd909d895964d52f73ffd1a"
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.925190    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm" event={"ID":"de1bfff9-9805-4b5a-a959-734a9a1570e8","Type":"ContainerDied","Data":"7597a98fd8dea289a2b603d0c2698dc016d4438f09825da411457313e13eb8b2"}
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.925309    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.928604    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h" event={"ID":"66456109-7df3-4de7-978e-13496801fff2","Type":"ContainerDied","Data":"e602d22a4538dcec1132463914cecd277a5d539b566898197d754b6e93c8c877"}
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.928710    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.937000    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9" event={"ID":"998580a2-a1bf-4eb6-b8db-b3277af46957","Type":"ContainerDied","Data":"0f8de1681ba801360fa150330262d97393105c0805c571173259328f2e80daad"}
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.937094    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.945754    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.945900    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp" event={"ID":"59ac7f13-0f86-43e3-9189-d89aa39defb9","Type":"ContainerDied","Data":"0d96afea2775b91be810259109a1f6b1c592375d00b8fac3f82582a4d4962c42"}
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.947820    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1bfff9-9805-4b5a-a959-734a9a1570e8-kube-api-access-xkrtw\" (UniqueName: \"kubernetes.io/projected/de1bfff9-9805-4b5a-a959-734a9a1570e8-kube-api-access-xkrtw\") pod \"de1bfff9-9805-4b5a-a959-734a9a1570e8\" (UID: \"de1bfff9-9805-4b5a-a959-734a9a1570e8\") "
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.948055    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1bfff9-9805-4b5a-a959-734a9a1570e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1bfff9-9805-4b5a-a959-734a9a1570e8-maas-api-service-ca\") pod \"de1bfff9-9805-4b5a-a959-734a9a1570e8\" (UID: \"de1bfff9-9805-4b5a-a959-734a9a1570e8\") "
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.948952    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998580a2-a1bf-4eb6-b8db-b3277af46957-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.948979    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0393d0-1fa2-4248-9d19-f31b08556afa-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.948995    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv696\" (UniqueName: \"kubernetes.io/projected/552e7cb2-282d-4f7e-b4ca-952b9df7a041-kube-api-access-wv696\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.949009    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8tq7\" (UniqueName: \"kubernetes.io/projected/998580a2-a1bf-4eb6-b8db-b3277af46957-kube-api-access-r8tq7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.949023    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k56z\" (UniqueName: \"kubernetes.io/projected/7f0393d0-1fa2-4248-9d19-f31b08556afa-kube-api-access-4k56z\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.949035    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e7cb2-282d-4f7e-b4ca-952b9df7a041-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.949843    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1bfff9-9805-4b5a-a959-734a9a1570e8-maas-api-service-ca" pod "de1bfff9-9805-4b5a-a959-734a9a1570e8" (UID: "de1bfff9-9805-4b5a-a959-734a9a1570e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.956410    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2" event={"ID":"f49c1907-0584-44be-8022-3b27cc76ec35","Type":"ContainerDied","Data":"ffac1c9fd365afa2b181ef4246dbb84597e7b02b897a89d5faed63c5111fd7f9"}
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.956623    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.967243    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.967190    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw" event={"ID":"ebf48e8e-316b-4250-9bae-9bcb736d3cb9","Type":"ContainerDied","Data":"d6c90317f298002b288b63ca1232de7744460f0140464e0e2ab1ddf4a5edf431"}
Oct 30 13:45:26.970647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.969538    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1bfff9-9805-4b5a-a959-734a9a1570e8-kube-api-access-xkrtw" pod "de1bfff9-9805-4b5a-a959-734a9a1570e8" (UID: "de1bfff9-9805-4b5a-a959-734a9a1570e8"). InnerVolumeSpecName "kube-api-access-xkrtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.003617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.976308    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"
Oct 30 13:45:27.003617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:26.976401    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj" event={"ID":"613f276e-cad4-40e0-84e8-b5c7e2524853","Type":"ContainerDied","Data":"d01ea1512817a792e3201cff75475dc60ef86bdd680eb78966a81705b7d70113"}
Oct 30 13:45:27.003617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.002098    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc" event={"ID":"75087b72-ac15-4d81-bd96-659ac0f7b2ef","Type":"ContainerDied","Data":"8dde1fbb68263fab505741949ae5c1f757888a422b72cdcafd5965f47834b7ab"}
Oct 30 13:45:27.003617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.002254    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"
Oct 30 13:45:27.007021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.006971    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb" event={"ID":"a7bc62a2-0765-497f-8529-8a11acfeb6be","Type":"ContainerDied","Data":"d1a5f6996523c9e80d74ba014df31780ef7b3a2ef811a7edf2c5975a14ced408"}
Oct 30 13:45:27.007165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.007115    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"
Oct 30 13:45:27.012392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.011372    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg" event={"ID":"552e7cb2-282d-4f7e-b4ca-952b9df7a041","Type":"ContainerDied","Data":"74e28f420a3d1254d51c5bc39a3c9960f933ad80e765f7def5ce6bf2c3258855"}
Oct 30 13:45:27.012392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.011491    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"
Oct 30 13:45:27.014428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.014407    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"
Oct 30 13:45:27.014589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.014452    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh" event={"ID":"816d95a5-15ed-418c-93f1-79b39600a5e3","Type":"ContainerDied","Data":"ddb92439175bbffc41543c5c2646b4a30539c98c13e106c04fb5f72af7c93504"}
Oct 30 13:45:27.025794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.025711    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2" event={"ID":"d8394c36-ec95-44a0-a6d3-a01716310ebc","Type":"ContainerDied","Data":"018918d3dcac43b45790a9b103a58ae7cdfea1ff060bab7a18503e9a9063a45f"}
Oct 30 13:45:27.026079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.026047    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"
Oct 30 13:45:27.051467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.051408    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1bfff9-9805-4b5a-a959-734a9a1570e8-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.051467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.051444    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrtw\" (UniqueName: \"kubernetes.io/projected/de1bfff9-9805-4b5a-a959-734a9a1570e8-kube-api-access-xkrtw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.110978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.110946    4993 scope.go:122] "RemoveContainer" containerID="3a2c462b2a4e70232d3a1c08431eafb408ce027475e656271aa60ae35777b6c7"
Oct 30 13:45:27.121385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.117729    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:27.150226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.145144    4993 scope.go:122] "RemoveContainer" containerID="7e1c9c5cb7ce53c265471e99abcb2a195233b4a19f8ac355879a3aba9a76ae05"
Oct 30 13:45:27.182417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.180048    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:27.221311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.214484    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:27.221311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.214914    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:27.227243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.223945    4993 scope.go:122] "RemoveContainer" containerID="f050d6f3c203a8c04522da8ee82cdedffce701fd7d68150404e5edc8470da2e0"
Oct 30 13:45:27.237914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.235708    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:27.238829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.238807    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:27.248081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.248054    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:27.263170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.263141    4993 scope.go:122] "RemoveContainer" containerID="f63269caea7adbef39ee6b58d02c56f3fddaf11cc1ba84cdeb3718bc46fa2fa3"
Oct 30 13:45:27.264005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.263978    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-kube-api-access-hjbw4\" (UniqueName: \"kubernetes.io/projected/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-kube-api-access-hjbw4\") pod \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\" (UID: \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\") "
Oct 30 13:45:27.264113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.264095    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2907709a-ad1d-4724-ab42-02ad7defb4b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2907709a-ad1d-4724-ab42-02ad7defb4b1-maas-api-service-ca\") pod \"2907709a-ad1d-4724-ab42-02ad7defb4b1\" (UID: \"2907709a-ad1d-4724-ab42-02ad7defb4b1\") "
Oct 30 13:45:27.264167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.264146    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-maas-api-service-ca\") pod \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\" (UID: \"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0\") "
Oct 30 13:45:27.264296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.264270    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2907709a-ad1d-4724-ab42-02ad7defb4b1-kube-api-access-6t8km\" (UniqueName: \"kubernetes.io/projected/2907709a-ad1d-4724-ab42-02ad7defb4b1-kube-api-access-6t8km\") pod \"2907709a-ad1d-4724-ab42-02ad7defb4b1\" (UID: \"2907709a-ad1d-4724-ab42-02ad7defb4b1\") "
Oct 30 13:45:27.266170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.266141    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2907709a-ad1d-4724-ab42-02ad7defb4b1-maas-api-service-ca" pod "2907709a-ad1d-4724-ab42-02ad7defb4b1" (UID: "2907709a-ad1d-4724-ab42-02ad7defb4b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:27.266861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.266833    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-maas-api-service-ca" pod "a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" (UID: "a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:27.269276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.269251    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2907709a-ad1d-4724-ab42-02ad7defb4b1-kube-api-access-6t8km" pod "2907709a-ad1d-4724-ab42-02ad7defb4b1" (UID: "2907709a-ad1d-4724-ab42-02ad7defb4b1"). InnerVolumeSpecName "kube-api-access-6t8km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.271589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.271563    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-kube-api-access-hjbw4" pod "a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" (UID: "a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0"). InnerVolumeSpecName "kube-api-access-hjbw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.336354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.336324    4993 scope.go:122] "RemoveContainer" containerID="6dc103bd0a942386317fd7e5ac2ab665824323bdcaa57901a6542a1ef887b632"
Oct 30 13:45:27.341185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.340475    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:27.365784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.365747    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-maas-api-service-ca\") pod \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\" (UID: \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\") "
Oct 30 13:45:27.366300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.366265    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380b156b-fe53-460f-aa79-2a9f98e37bd2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b156b-fe53-460f-aa79-2a9f98e37bd2-maas-api-service-ca\") pod \"380b156b-fe53-460f-aa79-2a9f98e37bd2\" (UID: \"380b156b-fe53-460f-aa79-2a9f98e37bd2\") "
Oct 30 13:45:27.366849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.366376    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-maas-api-service-ca" pod "97c724a9-ac88-4e9e-9d50-c8bf38d366a8" (UID: "97c724a9-ac88-4e9e-9d50-c8bf38d366a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:27.366849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.366756    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380b156b-fe53-460f-aa79-2a9f98e37bd2-maas-api-service-ca" pod "380b156b-fe53-460f-aa79-2a9f98e37bd2" (UID: "380b156b-fe53-460f-aa79-2a9f98e37bd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:27.367112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.367043    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-kube-api-access-jgx5z\" (UniqueName: \"kubernetes.io/projected/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-kube-api-access-jgx5z\") pod \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\" (UID: \"97c724a9-ac88-4e9e-9d50-c8bf38d366a8\") "
Oct 30 13:45:27.368133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.367356    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9325a541-f5aa-4272-bbe1-c75fd95e384a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a541-f5aa-4272-bbe1-c75fd95e384a-maas-api-service-ca\") pod \"9325a541-f5aa-4272-bbe1-c75fd95e384a\" (UID: \"9325a541-f5aa-4272-bbe1-c75fd95e384a\") "
Oct 30 13:45:27.368133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.367728    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9325a541-f5aa-4272-bbe1-c75fd95e384a-maas-api-service-ca" pod "9325a541-f5aa-4272-bbe1-c75fd95e384a" (UID: "9325a541-f5aa-4272-bbe1-c75fd95e384a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:27.368133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.368091    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7febc0c4-d800-4a15-b9b8-708d4517a176-maas-api-service-ca" pod "7febc0c4-d800-4a15-b9b8-708d4517a176" (UID: "7febc0c4-d800-4a15-b9b8-708d4517a176"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:27.368584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.368394    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7febc0c4-d800-4a15-b9b8-708d4517a176-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7febc0c4-d800-4a15-b9b8-708d4517a176-maas-api-service-ca\") pod \"7febc0c4-d800-4a15-b9b8-708d4517a176\" (UID: \"7febc0c4-d800-4a15-b9b8-708d4517a176\") "
Oct 30 13:45:27.369087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.368997    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7febc0c4-d800-4a15-b9b8-708d4517a176-kube-api-access-6zd82\" (UniqueName: \"kubernetes.io/projected/7febc0c4-d800-4a15-b9b8-708d4517a176-kube-api-access-6zd82\") pod \"7febc0c4-d800-4a15-b9b8-708d4517a176\" (UID: \"7febc0c4-d800-4a15-b9b8-708d4517a176\") "
Oct 30 13:45:27.370601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.370535    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9325a541-f5aa-4272-bbe1-c75fd95e384a-kube-api-access-jf2c6\" (UniqueName: \"kubernetes.io/projected/9325a541-f5aa-4272-bbe1-c75fd95e384a-kube-api-access-jf2c6\") pod \"9325a541-f5aa-4272-bbe1-c75fd95e384a\" (UID: \"9325a541-f5aa-4272-bbe1-c75fd95e384a\") "
Oct 30 13:45:27.371740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.371140    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380b156b-fe53-460f-aa79-2a9f98e37bd2-kube-api-access-ldtw9\" (UniqueName: \"kubernetes.io/projected/380b156b-fe53-460f-aa79-2a9f98e37bd2-kube-api-access-ldtw9\") pod \"380b156b-fe53-460f-aa79-2a9f98e37bd2\" (UID: \"380b156b-fe53-460f-aa79-2a9f98e37bd2\") "
Oct 30 13:45:27.374643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.374611    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjbw4\" (UniqueName: \"kubernetes.io/projected/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-kube-api-access-hjbw4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.374765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.374752    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2907709a-ad1d-4724-ab42-02ad7defb4b1-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.374920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.374895    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.375029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.375015    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t8km\" (UniqueName: \"kubernetes.io/projected/2907709a-ad1d-4724-ab42-02ad7defb4b1-kube-api-access-6t8km\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.375123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.375112    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.375217 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.375192    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b156b-fe53-460f-aa79-2a9f98e37bd2-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.375414 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.375365    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a541-f5aa-4272-bbe1-c75fd95e384a-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.375414 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.375385    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7febc0c4-d800-4a15-b9b8-708d4517a176-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.386822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.386182    4993 scope.go:122] "RemoveContainer" containerID="f5e219312b6331e77e4c4d1993181d9fdb208bd0438ab9de8dd115ea0844b2d1"
Oct 30 13:45:27.389691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.389621    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9325a541-f5aa-4272-bbe1-c75fd95e384a-kube-api-access-jf2c6" pod "9325a541-f5aa-4272-bbe1-c75fd95e384a" (UID: "9325a541-f5aa-4272-bbe1-c75fd95e384a"). InnerVolumeSpecName "kube-api-access-jf2c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.390012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.389971    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7febc0c4-d800-4a15-b9b8-708d4517a176-kube-api-access-6zd82" pod "7febc0c4-d800-4a15-b9b8-708d4517a176" (UID: "7febc0c4-d800-4a15-b9b8-708d4517a176"). InnerVolumeSpecName "kube-api-access-6zd82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.392577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.392531    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380b156b-fe53-460f-aa79-2a9f98e37bd2-kube-api-access-ldtw9" pod "380b156b-fe53-460f-aa79-2a9f98e37bd2" (UID: "380b156b-fe53-460f-aa79-2a9f98e37bd2"). InnerVolumeSpecName "kube-api-access-ldtw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.395489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.395466    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-kube-api-access-jgx5z" pod "97c724a9-ac88-4e9e-9d50-c8bf38d366a8" (UID: "97c724a9-ac88-4e9e-9d50-c8bf38d366a8"). InnerVolumeSpecName "kube-api-access-jgx5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.413259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.413228    4993 scope.go:122] "RemoveContainer" containerID="3f2ed0a405be7c795777799aaa58fb5733b815617e9643856111f9554b8e6256"
Oct 30 13:45:27.437228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.437191    4993 scope.go:122] "RemoveContainer" containerID="3a6c19da578c1987c97b78c3dd2791dc569ef799141f2ed38e2659f437f40057"
Oct 30 13:45:27.454230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.454173    4993 scope.go:122] "RemoveContainer" containerID="cb111761756dcab4012b532710867d4316e7c26090b495fd3a17077415501ee6"
Oct 30 13:45:27.474119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.474091    4993 scope.go:122] "RemoveContainer" containerID="82878e6eb276aea07f17616f9516fe7e458f7b357f0e88cd6729276789e2656f"
Oct 30 13:45:27.476199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.476178    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49589327-9884-4386-a522-df5b12d23094-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589327-9884-4386-a522-df5b12d23094-maas-api-service-ca\") pod \"49589327-9884-4386-a522-df5b12d23094\" (UID: \"49589327-9884-4386-a522-df5b12d23094\") "
Oct 30 13:45:27.476385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.476366    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49589327-9884-4386-a522-df5b12d23094-kube-api-access-gcqlb\" (UniqueName: \"kubernetes.io/projected/49589327-9884-4386-a522-df5b12d23094-kube-api-access-gcqlb\") pod \"49589327-9884-4386-a522-df5b12d23094\" (UID: \"49589327-9884-4386-a522-df5b12d23094\") "
Oct 30 13:45:27.477348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.477325    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zd82\" (UniqueName: \"kubernetes.io/projected/7febc0c4-d800-4a15-b9b8-708d4517a176-kube-api-access-6zd82\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.477474 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.477459    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf2c6\" (UniqueName: \"kubernetes.io/projected/9325a541-f5aa-4272-bbe1-c75fd95e384a-kube-api-access-jf2c6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.477598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.477582    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldtw9\" (UniqueName: \"kubernetes.io/projected/380b156b-fe53-460f-aa79-2a9f98e37bd2-kube-api-access-ldtw9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.477693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.477672    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49589327-9884-4386-a522-df5b12d23094-maas-api-service-ca" pod "49589327-9884-4386-a522-df5b12d23094" (UID: "49589327-9884-4386-a522-df5b12d23094"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:27.477791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.477680    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgx5z\" (UniqueName: \"kubernetes.io/projected/97c724a9-ac88-4e9e-9d50-c8bf38d366a8-kube-api-access-jgx5z\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.478636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.478609    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"]
Oct 30 13:45:27.479836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.479802    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49589327-9884-4386-a522-df5b12d23094-kube-api-access-gcqlb" pod "49589327-9884-4386-a522-df5b12d23094" (UID: "49589327-9884-4386-a522-df5b12d23094"). InnerVolumeSpecName "kube-api-access-gcqlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:27.480142 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:27.480111    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2bf8f0_097c_453e_8cd5_d1d3b94c8710.slice/crio-3039d4c05cc7accf550d672291ad1897b63067f6b1263af8f90131e6f7074be1 WatchSource:0}: Error finding container 3039d4c05cc7accf550d672291ad1897b63067f6b1263af8f90131e6f7074be1: Status 404 returned error can't find the container with id 3039d4c05cc7accf550d672291ad1897b63067f6b1263af8f90131e6f7074be1
Oct 30 13:45:27.495047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.495020    4993 scope.go:122] "RemoveContainer" containerID="e6efb607c326f618e638ec514ece569de2466365920978aa141a9a945dd52f22"
Oct 30 13:45:27.556903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.556876    4993 scope.go:122] "RemoveContainer" containerID="e2230eaa027005a14fc3294778abf4ed2cf2e5d535b965cee6f2ccc377c02877"
Oct 30 13:45:27.579548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.579458    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589327-9884-4386-a522-df5b12d23094-maas-api-service-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.579548 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.579491    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcqlb\" (UniqueName: \"kubernetes.io/projected/49589327-9884-4386-a522-df5b12d23094-kube-api-access-gcqlb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:27.614112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.614035    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"]
Oct 30 13:45:27.614112 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:27.614084    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1faa809_72e9_49a2_8fa4_e97d9de1a166.slice/crio-1ed3f1b727c2e074c250007f749178cdcca2755c56ee37f00956e0026aa141ec WatchSource:0}: Error finding container 1ed3f1b727c2e074c250007f749178cdcca2755c56ee37f00956e0026aa141ec: Status 404 returned error can't find the container with id 1ed3f1b727c2e074c250007f749178cdcca2755c56ee37f00956e0026aa141ec
Oct 30 13:45:27.697434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.697400    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2" path="/var/lib/kubelet/pods/3de2b6ea-dd19-4d6f-8b8d-1f5155cd6cb2/volumes"
Oct 30 13:45:27.698344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.697981    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96952ed-4208-4568-9005-6a53e3f36eb6" path="/var/lib/kubelet/pods/b96952ed-4208-4568-9005-6a53e3f36eb6/volumes"
Oct 30 13:45:27.698608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.698586    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b" path="/var/lib/kubelet/pods/fb7a5cbe-65aa-40a9-9f8e-ef4b1a19c41b/volumes"
Oct 30 13:45:27.823195 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.823155    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"]
Oct 30 13:45:27.828731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.828694    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f97234-29889465-bhfnw"]
Oct 30 13:45:27.955334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.955299    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"]
Oct 30 13:45:27.962170 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.962139    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0505b0-29889465-flklc"]
Oct 30 13:45:27.982507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.982479    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"]
Oct 30 13:45:27.990468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:27.990435    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889465-pbnwg"]
Oct 30 13:45:28.024188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.024146    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"]
Oct 30 13:45:28.038521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.038481    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1957dc-29889465-7zbkb"]
Oct 30 13:45:28.046162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.046134    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"
Oct 30 13:45:28.046345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.046130    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8" event={"ID":"9325a541-f5aa-4272-bbe1-c75fd95e384a","Type":"ContainerDied","Data":"a1de8fefcb78d2dc6e4a07f3596f4b46fbd655f8b38bc158ac46f487da7fd28f"}
Oct 30 13:45:28.046345 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.046315    4993 scope.go:122] "RemoveContainer" containerID="22aee46f66987a6bdd0e767455db5f68e1af24471fc2956658de417d12d118f3"
Oct 30 13:45:28.047814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.047789    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t" event={"ID":"a1faa809-72e9-49a2-8fa4-e97d9de1a166","Type":"ContainerStarted","Data":"1ed3f1b727c2e074c250007f749178cdcca2755c56ee37f00956e0026aa141ec"}
Oct 30 13:45:28.050845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.050819    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb" event={"ID":"a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0","Type":"ContainerDied","Data":"d36b4b37a1533697dfdab91dfb95e86fcc3d619564217e7a8eb5c65184c0e216"}
Oct 30 13:45:28.050976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.050861    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"
Oct 30 13:45:28.052616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.052596    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"
Oct 30 13:45:28.053154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.053128    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27" event={"ID":"97c724a9-ac88-4e9e-9d50-c8bf38d366a8","Type":"ContainerDied","Data":"37d2153c858f9328b2fa00c26aa57f1869cdd7deef6041cd4020d54d09348db0"}
Oct 30 13:45:28.067115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.067074    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"
Oct 30 13:45:28.068095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.067078    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz" event={"ID":"7febc0c4-d800-4a15-b9b8-708d4517a176","Type":"ContainerDied","Data":"7bb0d8f363f73cba680c6199317c29a2583f83c7171eadcd1e19ac746f239a23"}
Oct 30 13:45:28.068095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.067480    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"]
Oct 30 13:45:28.075269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.075246    4993 scope.go:122] "RemoveContainer" containerID="e92b702d2a162a07c8175e1c0ecf77871dd9bbadba69828c3a18ea4a50fa33e3"
Oct 30 13:45:28.078434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.078403    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm" event={"ID":"380b156b-fe53-460f-aa79-2a9f98e37bd2","Type":"ContainerDied","Data":"2435018a9855d02592fea085e824a4e548d7e0809692c9ee7ac423492ddecbea"}
Oct 30 13:45:28.078584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.078478    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"
Oct 30 13:45:28.080940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.080915    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09ad1e-29889465-jl8gm"]
Oct 30 13:45:28.082754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.082728    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg" event={"ID":"49589327-9884-4386-a522-df5b12d23094","Type":"ContainerDied","Data":"1d5c711a8d355d3700e9fd5ac0c26a667356433ca6747eada70fb1ef395178c5"}
Oct 30 13:45:28.082876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.082740    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"
Oct 30 13:45:28.094348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.093030    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg" event={"ID":"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710","Type":"ContainerStarted","Data":"5fc1b260bacf1f97e2451aa4c69539c2c9fcfd0fbbfac5560f1f730e666b93b3"}
Oct 30 13:45:28.094348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.093086    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg" event={"ID":"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710","Type":"ContainerStarted","Data":"3039d4c05cc7accf550d672291ad1897b63067f6b1263af8f90131e6f7074be1"}
Oct 30 13:45:28.095406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.095376    4993 scope.go:122] "RemoveContainer" containerID="937f5d1504abbeec4e909286182201ed9528c02c0c02b3f9f9703cbb4fb33f57"
Oct 30 13:45:28.103264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.103199    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"
Oct 30 13:45:28.116577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.116551    4993 scope.go:122] "RemoveContainer" containerID="f53c0970f587bbcce7d4d16b5630687c279ee9c5f7afe9cac34e375c6cd32760"
Oct 30 13:45:28.129275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.129190    4993 scope.go:122] "RemoveContainer" containerID="e7aa62bd5ced0c266c31e76d19500246f2275a748dfefd35dfc07bd9a8eb4055"
Oct 30 13:45:28.142326 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.142300    4993 scope.go:122] "RemoveContainer" containerID="4e8cec9321f333e7ca11beb78101888190228d2be6d35144025e473f67d1f97a"
Oct 30 13:45:28.150422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.150389    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"]
Oct 30 13:45:28.159383 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.159351    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-63499e-29889465-qm7f9"]
Oct 30 13:45:28.233687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.233643    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"]
Oct 30 13:45:28.238690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.238662    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889465-v5dp2"]
Oct 30 13:45:28.271630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.271594    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"]
Oct 30 13:45:28.282226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.282176    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889465-27vmh"]
Oct 30 13:45:28.323252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.323202    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"]
Oct 30 13:45:28.328647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.328614    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a8b38-29889465-frrl2"]
Oct 30 13:45:28.361053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.361012    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"]
Oct 30 13:45:28.366191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.366152    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9cc0e6-29889465-7xdf8"]
Oct 30 13:45:28.591347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.591299    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"]
Oct 30 13:45:28.598169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.598132    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-451a21-29889465-qm56h"]
Oct 30 13:45:28.628177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.628128    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"]
Oct 30 13:45:28.633251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.633199    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889465-45kmp"]
Oct 30 13:45:28.669728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.669677    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"]
Oct 30 13:45:28.675337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.675303    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7b7407-29889465-lbbgj"]
Oct 30 13:45:28.715202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.715161    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"]
Oct 30 13:45:28.724721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.724672    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8740d4-29889465-rbnsm"]
Oct 30 13:45:28.760283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.760246    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"]
Oct 30 13:45:28.769705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.769667    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889465-pqk27"]
Oct 30 13:45:28.798453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.798410    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"]
Oct 30 13:45:28.804025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.803988    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af8369-29889465-kjqm8"]
Oct 30 13:45:28.835484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.835442    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"]
Oct 30 13:45:28.840378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.840342    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-acf9e2-29889465-lxkqg"]
Oct 30 13:45:28.876087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.876010    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"]
Oct 30 13:45:28.884615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.884575    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889465-pjrc8"]
Oct 30 13:45:28.916457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.916388    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg" podStartSLOduration=4.916370877 podStartE2EDuration="4.916370877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:28.912520077 +0000 UTC m=+4393.388239689" watchObservedRunningTime="2026-10-30 13:45:28.916370877 +0000 UTC m=+4393.392090488"
Oct 30 13:45:28.953389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.953342    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"]
Oct 30 13:45:28.960800 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.960754    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889465-kkhzb"]
Oct 30 13:45:28.992248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.992200    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"]
Oct 30 13:45:28.999944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:28.999915    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889465-6jxdz"]
Oct 30 13:45:29.115566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.115528    4993 generic.go:356] "Generic (PLEG): container finished" podID="3a2bf8f0-097c-453e-8cd5-d1d3b94c8710" containerID="5fc1b260bacf1f97e2451aa4c69539c2c9fcfd0fbbfac5560f1f730e666b93b3" exitCode=0
Oct 30 13:45:29.115740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.115610    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg" event={"ID":"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710","Type":"ContainerDied","Data":"5fc1b260bacf1f97e2451aa4c69539c2c9fcfd0fbbfac5560f1f730e666b93b3"}
Oct 30 13:45:29.118688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.118659    4993 generic.go:356] "Generic (PLEG): container finished" podID="a1faa809-72e9-49a2-8fa4-e97d9de1a166" containerID="50b407247f68315930ca8184268a4e454ae2a2fb1f3b7bf01b8a992169aad713" exitCode=0
Oct 30 13:45:29.119157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.118737    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t" event={"ID":"a1faa809-72e9-49a2-8fa4-e97d9de1a166","Type":"ContainerDied","Data":"50b407247f68315930ca8184268a4e454ae2a2fb1f3b7bf01b8a992169aad713"}
Oct 30 13:45:29.696147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.696101    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" path="/var/lib/kubelet/pods/2907709a-ad1d-4724-ab42-02ad7defb4b1/volumes"
Oct 30 13:45:29.696744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.696717    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" path="/var/lib/kubelet/pods/380b156b-fe53-460f-aa79-2a9f98e37bd2/volumes"
Oct 30 13:45:29.697284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.697260    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49589327-9884-4386-a522-df5b12d23094" path="/var/lib/kubelet/pods/49589327-9884-4386-a522-df5b12d23094/volumes"
Oct 30 13:45:29.698089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.698064    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" path="/var/lib/kubelet/pods/552e7cb2-282d-4f7e-b4ca-952b9df7a041/volumes"
Oct 30 13:45:29.698591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.698570    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" path="/var/lib/kubelet/pods/59ac7f13-0f86-43e3-9189-d89aa39defb9/volumes"
Oct 30 13:45:29.699068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.699048    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" path="/var/lib/kubelet/pods/613f276e-cad4-40e0-84e8-b5c7e2524853/volumes"
Oct 30 13:45:29.699840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.699816    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66456109-7df3-4de7-978e-13496801fff2" path="/var/lib/kubelet/pods/66456109-7df3-4de7-978e-13496801fff2/volumes"
Oct 30 13:45:29.700287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.700272    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" path="/var/lib/kubelet/pods/75087b72-ac15-4d81-bd96-659ac0f7b2ef/volumes"
Oct 30 13:45:29.700681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.700668    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" path="/var/lib/kubelet/pods/7f0393d0-1fa2-4248-9d19-f31b08556afa/volumes"
Oct 30 13:45:29.701392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.701377    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" path="/var/lib/kubelet/pods/7febc0c4-d800-4a15-b9b8-708d4517a176/volumes"
Oct 30 13:45:29.701774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.701762    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" path="/var/lib/kubelet/pods/816d95a5-15ed-418c-93f1-79b39600a5e3/volumes"
Oct 30 13:45:29.702150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.702138    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" path="/var/lib/kubelet/pods/9325a541-f5aa-4272-bbe1-c75fd95e384a/volumes"
Oct 30 13:45:29.702832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.702817    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" path="/var/lib/kubelet/pods/97c724a9-ac88-4e9e-9d50-c8bf38d366a8/volumes"
Oct 30 13:45:29.703228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.703197    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" path="/var/lib/kubelet/pods/998580a2-a1bf-4eb6-b8db-b3277af46957/volumes"
Oct 30 13:45:29.703889 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.703873    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" path="/var/lib/kubelet/pods/a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0/volumes"
Oct 30 13:45:29.704319 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.704305    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" path="/var/lib/kubelet/pods/a7bc62a2-0765-497f-8529-8a11acfeb6be/volumes"
Oct 30 13:45:29.704708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.704696    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" path="/var/lib/kubelet/pods/d8394c36-ec95-44a0-a6d3-a01716310ebc/volumes"
Oct 30 13:45:29.705399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.705384    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" path="/var/lib/kubelet/pods/de1bfff9-9805-4b5a-a959-734a9a1570e8/volumes"
Oct 30 13:45:29.705783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.705770    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" path="/var/lib/kubelet/pods/ebf48e8e-316b-4250-9bae-9bcb736d3cb9/volumes"
Oct 30 13:45:29.706166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:29.706153    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f49c1907-0584-44be-8022-3b27cc76ec35" path="/var/lib/kubelet/pods/f49c1907-0584-44be-8022-3b27cc76ec35/volumes"
Oct 30 13:45:30.333588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.333559    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:30.373440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.373364    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-xks-ca\") pod \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") "
Oct 30 13:45:30.373647 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.373623    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1faa809-72e9-49a2-8fa4-e97d9de1a166-kube-api-access-c66wc\" (UniqueName: \"kubernetes.io/projected/a1faa809-72e9-49a2-8fa4-e97d9de1a166-kube-api-access-c66wc\") pod \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") "
Oct 30 13:45:30.373802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.373744    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-openshift-ca\") pod \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\" (UID: \"a1faa809-72e9-49a2-8fa4-e97d9de1a166\") "
Oct 30 13:45:30.374398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.374366    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-openshift-ca" pod "a1faa809-72e9-49a2-8fa4-e97d9de1a166" (UID: "a1faa809-72e9-49a2-8fa4-e97d9de1a166"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:30.374685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.374664    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.377437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.377382    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1faa809-72e9-49a2-8fa4-e97d9de1a166-kube-api-access-c66wc" pod "a1faa809-72e9-49a2-8fa4-e97d9de1a166" (UID: "a1faa809-72e9-49a2-8fa4-e97d9de1a166"). InnerVolumeSpecName "kube-api-access-c66wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:30.379128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.379101    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-xks-ca" pod "a1faa809-72e9-49a2-8fa4-e97d9de1a166" (UID: "a1faa809-72e9-49a2-8fa4-e97d9de1a166"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:30.383927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.383869    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:30.475749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.475708    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-openshift-ca\") pod \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") "
Oct 30 13:45:30.476062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.476023    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-xks-ca\") pod \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") "
Oct 30 13:45:30.476155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.476117    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-kube-api-access-l29jn\" (UniqueName: \"kubernetes.io/projected/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-kube-api-access-l29jn\") pod \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\" (UID: \"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710\") "
Oct 30 13:45:30.476655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.476612    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-openshift-ca" pod "3a2bf8f0-097c-453e-8cd5-d1d3b94c8710" (UID: "3a2bf8f0-097c-453e-8cd5-d1d3b94c8710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:30.477067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.477039    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1faa809-72e9-49a2-8fa4-e97d9de1a166-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.477236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.477071    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.477236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.477085    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66wc\" (UniqueName: \"kubernetes.io/projected/a1faa809-72e9-49a2-8fa4-e97d9de1a166-kube-api-access-c66wc\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.479275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.479222    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-kube-api-access-l29jn" pod "3a2bf8f0-097c-453e-8cd5-d1d3b94c8710" (UID: "3a2bf8f0-097c-453e-8cd5-d1d3b94c8710"). InnerVolumeSpecName "kube-api-access-l29jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:30.479430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.479292    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-xks-ca" pod "3a2bf8f0-097c-453e-8cd5-d1d3b94c8710" (UID: "3a2bf8f0-097c-453e-8cd5-d1d3b94c8710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:30.578858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.578812    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.578858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:30.578860    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l29jn\" (UniqueName: \"kubernetes.io/projected/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710-kube-api-access-l29jn\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.144861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.144827    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"
Oct 30 13:45:31.145051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.144860    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg" event={"ID":"3a2bf8f0-097c-453e-8cd5-d1d3b94c8710","Type":"ContainerDied","Data":"3039d4c05cc7accf550d672291ad1897b63067f6b1263af8f90131e6f7074be1"}
Oct 30 13:45:31.145051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.144913    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3039d4c05cc7accf550d672291ad1897b63067f6b1263af8f90131e6f7074be1"
Oct 30 13:45:31.146622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.146593    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t" event={"ID":"a1faa809-72e9-49a2-8fa4-e97d9de1a166","Type":"ContainerDied","Data":"1ed3f1b727c2e074c250007f749178cdcca2755c56ee37f00956e0026aa141ec"}
Oct 30 13:45:31.146622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.146615    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"
Oct 30 13:45:31.146799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.146641    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed3f1b727c2e074c250007f749178cdcca2755c56ee37f00956e0026aa141ec"
Oct 30 13:45:31.449794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.449747    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:45:31.504908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.504874    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-credential-socket\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.504931    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-data\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.504992    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-token\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-token\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.505091    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-podinfo\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.505123    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-certs\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.505269    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-envoy\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.505320    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00cfe34-6ca8-4a28-a921-968f1368fb10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00cfe34-6ca8-4a28-a921-968f1368fb10-istiod-ca-cert\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.505398    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-kube-api-access-sr6w6\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-kube-api-access-sr6w6\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.505483    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-socket\") pod \"d00cfe34-6ca8-4a28-a921-968f1368fb10\" (UID: \"d00cfe34-6ca8-4a28-a921-968f1368fb10\") "
Oct 30 13:45:31.505992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.505962    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-socket" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:31.506586 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.506302    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00cfe34-6ca8-4a28-a921-968f1368fb10-istiod-ca-cert" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:31.506586 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.506322    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.506586 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.506577    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-data" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:31.506951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.506893    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-credential-socket" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:31.508570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.508533    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-podinfo" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:31.508668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.508578    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-envoy" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:31.508668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.508595    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-kube-api-access-sr6w6" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "kube-api-access-sr6w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:31.509862 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.509826    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-certs" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:31.510447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.509866    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-token" pod "d00cfe34-6ca8-4a28-a921-968f1368fb10" (UID: "d00cfe34-6ca8-4a28-a921-968f1368fb10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:31.611241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.609952    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.611241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.609995    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.611241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.610009    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.611241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.610023    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00cfe34-6ca8-4a28-a921-968f1368fb10-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.611241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.610040    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr6w6\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-kube-api-access-sr6w6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.611241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.610055    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.611241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.610072    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:31.611241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:31.610086    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00cfe34-6ca8-4a28-a921-968f1368fb10-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.158394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.158287    4993 generic.go:356] "Generic (PLEG): container finished" podID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerID="3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94" exitCode=0
Oct 30 13:45:32.158394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.158347    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"
Oct 30 13:45:32.158394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.158364    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" event={"ID":"d00cfe34-6ca8-4a28-a921-968f1368fb10","Type":"ContainerDied","Data":"3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94"}
Oct 30 13:45:32.158681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.158439    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz" event={"ID":"d00cfe34-6ca8-4a28-a921-968f1368fb10","Type":"ContainerDied","Data":"6a1fc055151016069cc59f258ff5b4b91fbab64f14e16b83589eee0e84180b08"}
Oct 30 13:45:32.158681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.158470    4993 scope.go:122] "RemoveContainer" containerID="3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94"
Oct 30 13:45:32.173157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.173131    4993 scope.go:122] "RemoveContainer" containerID="445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c"
Oct 30 13:45:32.188606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.188329    4993 scope.go:122] "RemoveContainer" containerID="3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94"
Oct 30 13:45:32.188696 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:32.188668    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94\": container with ID starting with 3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94 not found: ID does not exist" containerID="3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94"
Oct 30 13:45:32.188759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.188703    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94"} err="failed to get container status \"3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94\": rpc error: code = NotFound desc = could not find container \"3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94\": container with ID starting with 3a152678db82c646f1ef31b5d22fb9697d308ba7bfc3d6aa037ef91536758f94 not found: ID does not exist"
Oct 30 13:45:32.188759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.188725    4993 scope.go:122] "RemoveContainer" containerID="445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c"
Oct 30 13:45:32.189114 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:32.189075    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c\": container with ID starting with 445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c not found: ID does not exist" containerID="445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c"
Oct 30 13:45:32.189297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.189114    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c"} err="failed to get container status \"445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c\": rpc error: code = NotFound desc = could not find container \"445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c\": container with ID starting with 445217ac5d6125042708d7c5c31bd93536f4a6dbd338ebc74da29e1e2da6f43c not found: ID does not exist"
Oct 30 13:45:32.212969 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.212937    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"]
Oct 30 13:45:32.224636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:32.224607    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-7nxzz"]
Oct 30 13:45:33.693799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:33.693753    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" path="/var/lib/kubelet/pods/d00cfe34-6ca8-4a28-a921-968f1368fb10/volumes"
Oct 30 13:45:36.561581 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:36.561514    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerName="main" containerID="cri-o://4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403" gracePeriod=45
Oct 30 13:45:37.131913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.131879    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:45:37.275418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275359    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-model-cache\") pod \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") "
Oct 30 13:45:37.276370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275448    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tmp-dir\") pod \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") "
Oct 30 13:45:37.276370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275517    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-home\") pod \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") "
Oct 30 13:45:37.276370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275617    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-kube-api-access-gt96r\" (UniqueName: \"kubernetes.io/projected/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-kube-api-access-gt96r\") pod \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") "
Oct 30 13:45:37.276370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275677    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tls-certs\") pod \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") "
Oct 30 13:45:37.276370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275751    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-model-cache" pod "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" (UID: "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:37.276370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275764    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tmp-dir" pod "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" (UID: "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:37.276370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275794    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-dshm\") pod \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\" (UID: \"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7\") "
Oct 30 13:45:37.276370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.275839    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-home" pod "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" (UID: "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:37.277578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.276597    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:37.277578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.276623    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:37.277578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.276635    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:37.278489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.278461    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-dshm" pod "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" (UID: "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:37.278611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.278452    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tls-certs" pod "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" (UID: "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:37.278611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.278499    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-kube-api-access-gt96r" pod "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" (UID: "2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7"). InnerVolumeSpecName "kube-api-access-gt96r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:37.293045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.293006    4993 generic.go:356] "Generic (PLEG): container finished" podID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerID="4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403" exitCode=0
Oct 30 13:45:37.293271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.293073    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"
Oct 30 13:45:37.293355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.293286    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" event={"ID":"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7","Type":"ContainerDied","Data":"4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403"}
Oct 30 13:45:37.293355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.293334    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" event={"ID":"2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7","Type":"ContainerDied","Data":"702857cc736e12e50171de8dea6f0d180081b40c8c1a93223764b2b5a8146202"}
Oct 30 13:45:37.293470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.293360    4993 scope.go:122] "RemoveContainer" containerID="4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403"
Oct 30 13:45:37.321866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.317750    4993 scope.go:122] "RemoveContainer" containerID="4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403"
Oct 30 13:45:37.321866 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:37.318362    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403\": container with ID starting with 4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403 not found: ID does not exist" containerID="4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403"
Oct 30 13:45:37.321866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.318402    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403"} err="failed to get container status \"4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403\": rpc error: code = NotFound desc = could not find container \"4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403\": container with ID starting with 4d488b81fc3707895055eee5f82217e31c99c8d02b1b423b6e41251d8a1cd403 not found: ID does not exist"
Oct 30 13:45:37.362133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.362041    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"]
Oct 30 13:45:37.377638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.377600    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt96r\" (UniqueName: \"kubernetes.io/projected/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-kube-api-access-gt96r\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:37.377638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.377635    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:37.377638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.377646    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:37.391416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.391368    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm"]
Oct 30 13:45:37.693361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:37.693266    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" path="/var/lib/kubelet/pods/2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7/volumes"
Oct 30 13:45:38.101231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:38.098740    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f97234/test-model-f97234-kserve-5969dd6cc-2pvsm" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.54:8000/ready\": context deadline exceeded"
Oct 30 13:45:38.909061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:38.904424    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"]
Oct 30 13:45:38.919302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:38.919262    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9cc0e6-jk8cr"]
Oct 30 13:45:38.931375 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:38.931323    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:45:39.694305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:39.694270    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137eb5d2-b8e8-4a7a-af89-1983f66d446c" path="/var/lib/kubelet/pods/137eb5d2-b8e8-4a7a-af89-1983f66d446c/volumes"
Oct 30 13:45:46.350735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.350694    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"]
Oct 30 13:45:46.351294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.350994    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" containerID="cri-o://bd39b584f5621c755913a71aa7aa14d14ddeb82cf201406323babb186d373af6" gracePeriod=30
Oct 30 13:45:46.978714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.978669    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"]
Oct 30 13:45:46.979297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979280    4993 cpu_manager.go:414] "containerMap: removing container" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:46.979297 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979297    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979310    4993 cpu_manager.go:414] "containerMap: removing container" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979316    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979324    4993 cpu_manager.go:414] "containerMap: removing container" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979328    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979334    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979339    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979350    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979354    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979362    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerName="main"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979366    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerName="main"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979376    4993 cpu_manager.go:414] "containerMap: removing container" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979380    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979387    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979401    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979408    4993 cpu_manager.go:414] "containerMap: removing container" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979413    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979421    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979425    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979434    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979438    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979445    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:46.979452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979449    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979458    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979477    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979484    4993 cpu_manager.go:414] "containerMap: removing container" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979488    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979497    4993 cpu_manager.go:414] "containerMap: removing container" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979501    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979507    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979512    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979517    4993 cpu_manager.go:414] "containerMap: removing container" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979522    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979531    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979535    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979543    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979548    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979555    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979559    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979568    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979572    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979578    4993 cpu_manager.go:414] "containerMap: removing container" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979582    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979592    4993 cpu_manager.go:414] "containerMap: removing container" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979597    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979603    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979608    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979613    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979618    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979626    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979631    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979638    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979642    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979650    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3a2bf8f0-097c-453e-8cd5-d1d3b94c8710" containerName="revoke-keys"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979655    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2bf8f0-097c-453e-8cd5-d1d3b94c8710" containerName="revoke-keys"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979661    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979666    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979673    4993 cpu_manager.go:414] "containerMap: removing container" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979677    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979685    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979690    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979697    4993 cpu_manager.go:414] "containerMap: removing container" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979702    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979708    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979713    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979721    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979725    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979738    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a1faa809-72e9-49a2-8fa4-e97d9de1a166" containerName="revoke-keys"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979742    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1faa809-72e9-49a2-8fa4-e97d9de1a166" containerName="revoke-keys"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979750    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979754    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979762    4993 cpu_manager.go:414] "containerMap: removing container" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979766    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979771    4993 cpu_manager.go:414] "containerMap: removing container" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979776    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979782    4993 cpu_manager.go:414] "containerMap: removing container" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979786    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979794    4993 cpu_manager.go:414] "containerMap: removing container" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979800    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979899    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979908    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979915    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979921    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979927    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979932    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2bf8f0-097c-453e-8cd5-d1d3b94c8710" containerName="revoke-keys"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979937    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979941    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979948    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979952    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="613f276e-cad4-40e0-84e8-b5c7e2524853" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979958    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00cfe34-6ca8-4a28-a921-968f1368fb10" containerName="istio-proxy"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979964    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979968    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="66456109-7df3-4de7-978e-13496801fff2" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979973    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7bc62a2-0765-497f-8529-8a11acfeb6be" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979980    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979985    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979991    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1faa809-72e9-49a2-8fa4-e97d9de1a166" containerName="revoke-keys"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.979998    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980003    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980009    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980013    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1bfff9-9805-4b5a-a959-734a9a1570e8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980020    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980025    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980031    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ac7f13-0f86-43e3-9189-d89aa39defb9" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980039    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980044    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980049    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980056    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980061    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4a305e-72ba-4da6-bb86-d0ac5dfbb8a7" containerName="main"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980067    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980074    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980081    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980087    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="75087b72-ac15-4d81-bd96-659ac0f7b2ef" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980093    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2907709a-ad1d-4724-ab42-02ad7defb4b1" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980098    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980105    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980110    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980117    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980124    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980130    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf48e8e-316b-4250-9bae-9bcb736d3cb9" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980135    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:46.981221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.980142    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="552e7cb2-282d-4f7e-b4ca-952b9df7a041" containerName="cleanup"
Oct 30 13:45:46.988866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:46.985148    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.003121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.003083    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:45:47.014806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.014769    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:45:47.021968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.021868    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"]
Oct 30 13:45:47.022808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.022776    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.023135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.023105    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmjp\" (UniqueName: \"kubernetes.io/projected/7ef7ff44-9b68-448a-825c-8e92cb0d7961-kube-api-access-kdmjp\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.023342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.023319    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.026580 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.026544    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:45:47.026740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.026581    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:45:47.124568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.124521    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmjp\" (UniqueName: \"kubernetes.io/projected/7ef7ff44-9b68-448a-825c-8e92cb0d7961-kube-api-access-kdmjp\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.124754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.124633    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.124754 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.124701    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.125523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.125491    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.127362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.127333    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.151985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.151946    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmjp\" (UniqueName: \"kubernetes.io/projected/7ef7ff44-9b68-448a-825c-8e92cb0d7961-kube-api-access-kdmjp\") pod \"maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.261178 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.261081    4993 patch_prober.go:29] interesting pod/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:45:47.261178 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.261149    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:45:47.301100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.301054    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:47.636058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:47.636026    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"]
Oct 30 13:45:47.637080 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:47.637042    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef7ff44_9b68_448a_825c_8e92cb0d7961.slice/crio-9eaee6a89336391ae7da795c51294a892732e9905ec49b95e7b94e49bb2aeda3 WatchSource:0}: Error finding container 9eaee6a89336391ae7da795c51294a892732e9905ec49b95e7b94e49bb2aeda3: Status 404 returned error can't find the container with id 9eaee6a89336391ae7da795c51294a892732e9905ec49b95e7b94e49bb2aeda3
Oct 30 13:45:48.490823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:48.490778    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv" event={"ID":"7ef7ff44-9b68-448a-825c-8e92cb0d7961","Type":"ContainerStarted","Data":"949b2eaad0bfb3010de2bfd7eab089151dc5bfc202adf388d4f52eb49418ff2f"}
Oct 30 13:45:48.491017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:48.490846    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv" event={"ID":"7ef7ff44-9b68-448a-825c-8e92cb0d7961","Type":"ContainerStarted","Data":"9eaee6a89336391ae7da795c51294a892732e9905ec49b95e7b94e49bb2aeda3"}
Oct 30 13:45:48.541627 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:48.541557    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv" podStartSLOduration=2.541542413 podStartE2EDuration="2.541542413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:48.537480151 +0000 UTC m=+4413.013199772" watchObservedRunningTime="2026-10-30 13:45:48.541542413 +0000 UTC m=+4413.017262022"
Oct 30 13:45:49.225739 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:49.225694    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"]
Oct 30 13:45:49.226278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:49.225981    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" containerID="cri-o://45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b" gracePeriod=30
Oct 30 13:45:49.560457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:49.558392    4993 generic.go:356] "Generic (PLEG): container finished" podID="7ef7ff44-9b68-448a-825c-8e92cb0d7961" containerID="949b2eaad0bfb3010de2bfd7eab089151dc5bfc202adf388d4f52eb49418ff2f" exitCode=0
Oct 30 13:45:49.560457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:49.558499    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv" event={"ID":"7ef7ff44-9b68-448a-825c-8e92cb0d7961","Type":"ContainerDied","Data":"949b2eaad0bfb3010de2bfd7eab089151dc5bfc202adf388d4f52eb49418ff2f"}
Oct 30 13:45:50.006026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.005984    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"]
Oct 30 13:45:50.006738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006712    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:50.006738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006738    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9325a541-f5aa-4272-bbe1-c75fd95e384a" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006766    4993 cpu_manager.go:414] "containerMap: removing container" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006775    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006799    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006806    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006815    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006822    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006856    4993 cpu_manager.go:414] "containerMap: removing container" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006864    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006882    4993 cpu_manager.go:414] "containerMap: removing container" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006887    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006900    4993 cpu_manager.go:414] "containerMap: removing container" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:50.006917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.006904    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998580a2-a1bf-4eb6-b8db-b3277af46957" containerName="cleanup"
Oct 30 13:45:50.007304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.007024    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c00bd4-f3cd-413f-8e54-7e7c98db5ca0" containerName="cleanup"
Oct 30 13:45:50.007304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.007036    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0393d0-1fa2-4248-9d19-f31b08556afa" containerName="cleanup"
Oct 30 13:45:50.007304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.007047    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8394c36-ec95-44a0-a6d3-a01716310ebc" containerName="cleanup"
Oct 30 13:45:50.007304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.007053    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="380b156b-fe53-460f-aa79-2a9f98e37bd2" containerName="cleanup"
Oct 30 13:45:50.007304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.007060    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="49589327-9884-4386-a522-df5b12d23094" containerName="cleanup"
Oct 30 13:45:50.007304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.007069    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c724a9-ac88-4e9e-9d50-c8bf38d366a8" containerName="cleanup"
Oct 30 13:45:50.007304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.007076    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="816d95a5-15ed-418c-93f1-79b39600a5e3" containerName="cleanup"
Oct 30 13:45:50.007304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.007083    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7febc0c4-d800-4a15-b9b8-708d4517a176" containerName="cleanup"
Oct 30 13:45:50.010712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.010691    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.043095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.043050    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"]
Oct 30 13:45:50.054578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.054536    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"]
Oct 30 13:45:50.077620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.077367    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d3c88b5-jj6hd"]
Oct 30 13:45:50.081146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.080240    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.081146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.080351    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.081146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.080394    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7svw\" (UniqueName: \"kubernetes.io/projected/b884ff9b-adfa-4c81-8282-09def8b358a8-kube-api-access-g7svw\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.125926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.125874    4993 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:45:50.126143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.125954    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:45:50.182270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.182186    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.182442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.182302    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.182442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.182335    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7svw\" (UniqueName: \"kubernetes.io/projected/b884ff9b-adfa-4c81-8282-09def8b358a8-kube-api-access-g7svw\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.182999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.182974    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.185083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.185051    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.207299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.207264    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7svw\" (UniqueName: \"kubernetes.io/projected/b884ff9b-adfa-4c81-8282-09def8b358a8-kube-api-access-g7svw\") pod \"maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.326133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.326023    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:50.742393 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:45:50.742353    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb884ff9b_adfa_4c81_8282_09def8b358a8.slice/crio-262ae98e05fa866ff6b8e4a18f4080b3dfa6e75c9ac924c2476d6e12c1479c33 WatchSource:0}: Error finding container 262ae98e05fa866ff6b8e4a18f4080b3dfa6e75c9ac924c2476d6e12c1479c33: Status 404 returned error can't find the container with id 262ae98e05fa866ff6b8e4a18f4080b3dfa6e75c9ac924c2476d6e12c1479c33
Oct 30 13:45:50.747990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.747957    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"]
Oct 30 13:45:50.756031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.756002    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:50.808265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.808184    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef7ff44-9b68-448a-825c-8e92cb0d7961-kube-api-access-kdmjp\" (UniqueName: \"kubernetes.io/projected/7ef7ff44-9b68-448a-825c-8e92cb0d7961-kube-api-access-kdmjp\") pod \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") "
Oct 30 13:45:50.808473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.808374    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-xks-ca\") pod \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") "
Oct 30 13:45:50.808536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.808476    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-openshift-ca\") pod \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\" (UID: \"7ef7ff44-9b68-448a-825c-8e92cb0d7961\") "
Oct 30 13:45:50.809837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.809802    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-openshift-ca" pod "7ef7ff44-9b68-448a-825c-8e92cb0d7961" (UID: "7ef7ff44-9b68-448a-825c-8e92cb0d7961"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:50.811284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.811194    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef7ff44-9b68-448a-825c-8e92cb0d7961-kube-api-access-kdmjp" pod "7ef7ff44-9b68-448a-825c-8e92cb0d7961" (UID: "7ef7ff44-9b68-448a-825c-8e92cb0d7961"). InnerVolumeSpecName "kube-api-access-kdmjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:50.811423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.811317    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-xks-ca" pod "7ef7ff44-9b68-448a-825c-8e92cb0d7961" (UID: "7ef7ff44-9b68-448a-825c-8e92cb0d7961"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:50.910655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.910543    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdmjp\" (UniqueName: \"kubernetes.io/projected/7ef7ff44-9b68-448a-825c-8e92cb0d7961-kube-api-access-kdmjp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:50.910655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.910586    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:50.910655 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:50.910602    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7ff44-9b68-448a-825c-8e92cb0d7961-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.583096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.582850    4993 generic.go:356] "Generic (PLEG): container finished" podID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerID="bd39b584f5621c755913a71aa7aa14d14ddeb82cf201406323babb186d373af6" exitCode=0
Oct 30 13:45:51.583096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.582989    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" event={"ID":"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9","Type":"ContainerDied","Data":"bd39b584f5621c755913a71aa7aa14d14ddeb82cf201406323babb186d373af6"}
Oct 30 13:45:51.583096 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.583044    4993 scope.go:122] "RemoveContainer" containerID="255cf86de66be1b33cc2ba702c6e5b108f9c02f9be9c4db62b4e7573b760d94f"
Oct 30 13:45:51.586848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.586815    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv" event={"ID":"7ef7ff44-9b68-448a-825c-8e92cb0d7961","Type":"ContainerDied","Data":"9eaee6a89336391ae7da795c51294a892732e9905ec49b95e7b94e49bb2aeda3"}
Oct 30 13:45:51.587047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.587027    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eaee6a89336391ae7da795c51294a892732e9905ec49b95e7b94e49bb2aeda3"
Oct 30 13:45:51.587151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.587090    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"
Oct 30 13:45:51.596333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.596292    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6" event={"ID":"b884ff9b-adfa-4c81-8282-09def8b358a8","Type":"ContainerStarted","Data":"c79fef8bb2207b3aadb1e53dc5c403528cbc60be6af66fa54da90fd798fb0465"}
Oct 30 13:45:51.596521 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.596362    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6" event={"ID":"b884ff9b-adfa-4c81-8282-09def8b358a8","Type":"ContainerStarted","Data":"262ae98e05fa866ff6b8e4a18f4080b3dfa6e75c9ac924c2476d6e12c1479c33"}
Oct 30 13:45:51.676530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.676497    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:45:51.693195 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.693160    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037c4482-a6f7-450d-98a1-23300038fc4a" path="/var/lib/kubelet/pods/037c4482-a6f7-450d-98a1-23300038fc4a/volumes"
Oct 30 13:45:51.729065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.729008    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6" podStartSLOduration=2.728991916 podStartE2EDuration="2.728991916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:51.646914781 +0000 UTC m=+4416.122634405" watchObservedRunningTime="2026-10-30 13:45:51.728991916 +0000 UTC m=+4416.204711536"
Oct 30 13:45:51.826481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826366    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-envoy\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.826481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826432    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-kube-api-access-mgcdc\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-kube-api-access-mgcdc\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.826733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826509    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-certs\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.826733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826584    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-data\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.826733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826636    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istiod-ca-cert\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.826893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826746    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-credential-socket\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.827114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826877    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-certs" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:51.827114 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.827112    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-podinfo\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.827343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826893    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-data" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:51.827343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.826995    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-credential-socket" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:51.827343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.827052    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istiod-ca-cert" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:51.827343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.827263    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-socket\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.827585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.827363    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-token\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-token\") pod \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\" (UID: \"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9\") "
Oct 30 13:45:51.827585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.827574    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-socket" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:51.828134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.828108    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.828289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.828143    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.828289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.828159    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.828289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.828174    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.828289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.828187    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.853098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.853028    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-podinfo" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:51.853098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.853044    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-token" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:51.853098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.853062    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-envoy" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:51.853338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.853106    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-kube-api-access-mgcdc" pod "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" (UID: "3a0ce3b1-2a81-4f31-a099-23bfae6a27a9"). InnerVolumeSpecName "kube-api-access-mgcdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:51.929544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.929501    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.929544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.929542    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.929544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.929552    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:51.929794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:51.929563    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgcdc\" (UniqueName: \"kubernetes.io/projected/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9-kube-api-access-mgcdc\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:52.666241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.662764    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx" event={"ID":"3a0ce3b1-2a81-4f31-a099-23bfae6a27a9","Type":"ContainerDied","Data":"7a314cb11ef3169ee03d4efef0aafd8bd1e9708bbefa583deb920d04286d1188"}
Oct 30 13:45:52.666241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.662843    4993 scope.go:122] "RemoveContainer" containerID="bd39b584f5621c755913a71aa7aa14d14ddeb82cf201406323babb186d373af6"
Oct 30 13:45:52.666241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.662973    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"
Oct 30 13:45:52.690172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.689847    4993 generic.go:356] "Generic (PLEG): container finished" podID="b884ff9b-adfa-4c81-8282-09def8b358a8" containerID="c79fef8bb2207b3aadb1e53dc5c403528cbc60be6af66fa54da90fd798fb0465" exitCode=0
Oct 30 13:45:52.690172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.690023    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6" event={"ID":"b884ff9b-adfa-4c81-8282-09def8b358a8","Type":"ContainerDied","Data":"c79fef8bb2207b3aadb1e53dc5c403528cbc60be6af66fa54da90fd798fb0465"}
Oct 30 13:45:52.839104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.839066    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"]
Oct 30 13:45:52.844329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.844293    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9cc0e6-openshift-default-5459db5bff-gmgvx"]
Oct 30 13:45:52.962838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.962745    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"]
Oct 30 13:45:52.973703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:52.973662    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-09ad1e-bvfmg"]
Oct 30 13:45:53.694783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:53.694739    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" path="/var/lib/kubelet/pods/3a0ce3b1-2a81-4f31-a099-23bfae6a27a9/volumes"
Oct 30 13:45:53.695538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:53.695519    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2bf8f0-097c-453e-8cd5-d1d3b94c8710" path="/var/lib/kubelet/pods/3a2bf8f0-097c-453e-8cd5-d1d3b94c8710/volumes"
Oct 30 13:45:53.887138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:53.887098    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:54.028925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.028879    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b884ff9b-adfa-4c81-8282-09def8b358a8-kube-api-access-g7svw\" (UniqueName: \"kubernetes.io/projected/b884ff9b-adfa-4c81-8282-09def8b358a8-kube-api-access-g7svw\") pod \"b884ff9b-adfa-4c81-8282-09def8b358a8\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") "
Oct 30 13:45:54.029150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.029016    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-openshift-ca\") pod \"b884ff9b-adfa-4c81-8282-09def8b358a8\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") "
Oct 30 13:45:54.029275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.029145    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-xks-ca\") pod \"b884ff9b-adfa-4c81-8282-09def8b358a8\" (UID: \"b884ff9b-adfa-4c81-8282-09def8b358a8\") "
Oct 30 13:45:54.029523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.029495    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-openshift-ca" pod "b884ff9b-adfa-4c81-8282-09def8b358a8" (UID: "b884ff9b-adfa-4c81-8282-09def8b358a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:54.029819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.029799    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.031291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.031263    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b884ff9b-adfa-4c81-8282-09def8b358a8-kube-api-access-g7svw" pod "b884ff9b-adfa-4c81-8282-09def8b358a8" (UID: "b884ff9b-adfa-4c81-8282-09def8b358a8"). InnerVolumeSpecName "kube-api-access-g7svw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:54.031403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.031371    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-xks-ca" pod "b884ff9b-adfa-4c81-8282-09def8b358a8" (UID: "b884ff9b-adfa-4c81-8282-09def8b358a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:54.131483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.131437    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7svw\" (UniqueName: \"kubernetes.io/projected/b884ff9b-adfa-4c81-8282-09def8b358a8-kube-api-access-g7svw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.131483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.131480    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b884ff9b-adfa-4c81-8282-09def8b358a8-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.504458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.504428    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:45:54.640277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640118    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-certs\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.640277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640184    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-envoy\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.640482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640322    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istiod-ca-cert\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.640482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640386    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-podinfo\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.640482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640432    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-socket\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.640482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640461    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-data\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.640618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640510    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-certs" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:54.640618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640530    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-token\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-token\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.640618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640587    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-kube-api-access-9x2mt\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-kube-api-access-9x2mt\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.640710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.640635    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-credential-socket\") pod \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\" (UID: \"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30\") "
Oct 30 13:45:54.641041 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.641014    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-data" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:54.641467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.641443    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.641593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.641551    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.641691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.641492    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-socket" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:54.641691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.641509    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-credential-socket" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:54.641691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.641643    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istiod-ca-cert" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:54.643573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.643546    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-token" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:54.643805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.643781    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-envoy" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:54.643880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.643822    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-podinfo" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:54.643986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.643962    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-kube-api-access-9x2mt" pod "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" (UID: "b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30"). InnerVolumeSpecName "kube-api-access-9x2mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:54.711951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.711916    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"
Oct 30 13:45:54.712542 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.711915    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6" event={"ID":"b884ff9b-adfa-4c81-8282-09def8b358a8","Type":"ContainerDied","Data":"262ae98e05fa866ff6b8e4a18f4080b3dfa6e75c9ac924c2476d6e12c1479c33"}
Oct 30 13:45:54.712542 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.712058    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262ae98e05fa866ff6b8e4a18f4080b3dfa6e75c9ac924c2476d6e12c1479c33"
Oct 30 13:45:54.714751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.714716    4993 generic.go:356] "Generic (PLEG): container finished" podID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerID="45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b" exitCode=0
Oct 30 13:45:54.714859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.714767    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"
Oct 30 13:45:54.714859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.714773    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" event={"ID":"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30","Type":"ContainerDied","Data":"45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b"}
Oct 30 13:45:54.714859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.714827    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr" event={"ID":"b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30","Type":"ContainerDied","Data":"015de7fb2fae1906e8ab6ffeafdd086db9f48f268997c395d3fc8f17798fb8e6"}
Oct 30 13:45:54.714859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.714848    4993 scope.go:122] "RemoveContainer" containerID="45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b"
Oct 30 13:45:54.731124 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.731097    4993 scope.go:122] "RemoveContainer" containerID="bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873"
Oct 30 13:45:54.743015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.742981    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.743015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.743015    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.743266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.743027    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.743266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.743039    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.743266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.743051    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.743266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.743065    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x2mt\" (UniqueName: \"kubernetes.io/projected/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-kube-api-access-9x2mt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.743266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.743078    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.746284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.746255    4993 scope.go:122] "RemoveContainer" containerID="45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b"
Oct 30 13:45:54.746665 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:54.746644    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b\": container with ID starting with 45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b not found: ID does not exist" containerID="45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b"
Oct 30 13:45:54.746755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.746679    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b"} err="failed to get container status \"45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b\": rpc error: code = NotFound desc = could not find container \"45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b\": container with ID starting with 45cee00a1ee04c1e21a8ae6f8a237bfd892092dea64d3ca3f4b0aeb3e5d8bf7b not found: ID does not exist"
Oct 30 13:45:54.746755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.746703    4993 scope.go:122] "RemoveContainer" containerID="bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873"
Oct 30 13:45:54.746999 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:45:54.746980    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873\": container with ID starting with bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873 not found: ID does not exist" containerID="bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873"
Oct 30 13:45:54.747053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.747007    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873"} err="failed to get container status \"bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873\": rpc error: code = NotFound desc = could not find container \"bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873\": container with ID starting with bf527b548f0d032d9ec583447bd8e7049617079b1cc29bef83c6457311721873 not found: ID does not exist"
Oct 30 13:45:54.772287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.772162    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"]
Oct 30 13:45:54.776944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:54.776896    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-hwqxr"]
Oct 30 13:45:55.694480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:45:55.694437    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" path="/var/lib/kubelet/pods/b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30/volumes"
Oct 30 13:46:01.292232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.291542    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"]
Oct 30 13:46:01.292232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.291823    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" containerID="cri-o://70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86" gracePeriod=30
Oct 30 13:46:01.904481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.904441    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"]
Oct 30 13:46:01.905080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905066    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b884ff9b-adfa-4c81-8282-09def8b358a8" containerName="revoke-keys"
Oct 30 13:46:01.905127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905082    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b884ff9b-adfa-4c81-8282-09def8b358a8" containerName="revoke-keys"
Oct 30 13:46:01.905127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905097    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7ef7ff44-9b68-448a-825c-8e92cb0d7961" containerName="revoke-keys"
Oct 30 13:46:01.905127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905103    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef7ff44-9b68-448a-825c-8e92cb0d7961" containerName="revoke-keys"
Oct 30 13:46:01.905127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905121    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy"
Oct 30 13:46:01.905127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905128    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905147    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905153    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905161    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905166    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905278    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905294    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b884ff9b-adfa-4c81-8282-09def8b358a8" containerName="revoke-keys"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905302    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef7ff44-9b68-448a-825c-8e92cb0d7961" containerName="revoke-keys"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905313    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy"
Oct 30 13:46:01.905362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.905320    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0ce3b1-2a81-4f31-a099-23bfae6a27a9" containerName="istio-proxy"
Oct 30 13:46:01.910716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.910691    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:01.930964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.930922    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"]
Oct 30 13:46:01.938015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.937978    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:46:01.938015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.938002    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:46:01.938015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.937998    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:46:01.938352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:01.938036    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:46:02.046179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.046126    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlqz2\" (UniqueName: \"kubernetes.io/projected/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-kube-api-access-tlqz2\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.046402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.046375    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.046515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.046495    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.046660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.046643    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"]
Oct 30 13:46:02.051287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.051249    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783efb0-n6xbb"]
Oct 30 13:46:02.148332 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.148285    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.148532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.148393    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.148532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.148465    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlqz2\" (UniqueName: \"kubernetes.io/projected/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-kube-api-access-tlqz2\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.149058 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.149032    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.150997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.150973    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.175685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.175608    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlqz2\" (UniqueName: \"kubernetes.io/projected/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-kube-api-access-tlqz2\") pod \"maas-api-revoke-keys-e2e-inf-b-af8369-q82jt\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.225757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.225719    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:02.566427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.566383    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"]
Oct 30 13:46:02.965608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:02.965567    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt" event={"ID":"8a0f31fe-5f6f-435e-b794-7a80f39c76c5","Type":"ContainerStarted","Data":"f66d1a0b132a0e16b390e22ed30efafcd92257ca9127d2cfb701fc14289b6156"}
Oct 30 13:46:03.693430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:03.693389    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d" path="/var/lib/kubelet/pods/7f58a8f3-d94a-4ac9-884f-cc8dd4eb188d/volumes"
Oct 30 13:46:03.977457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:03.977416    4993 generic.go:356] "Generic (PLEG): container finished" podID="8a0f31fe-5f6f-435e-b794-7a80f39c76c5" containerID="13a919a357d6886d8af4c9acc133c0b61a91ee84db8672385ffc65ed64b945ae" exitCode=0
Oct 30 13:46:03.977644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:03.977496    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt" event={"ID":"8a0f31fe-5f6f-435e-b794-7a80f39c76c5","Type":"ContainerDied","Data":"13a919a357d6886d8af4c9acc133c0b61a91ee84db8672385ffc65ed64b945ae"}
Oct 30 13:46:05.165231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.165174    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:05.194247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.194186    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-xks-ca\") pod \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") "
Oct 30 13:46:05.194497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.194323    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-kube-api-access-tlqz2\" (UniqueName: \"kubernetes.io/projected/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-kube-api-access-tlqz2\") pod \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") "
Oct 30 13:46:05.194497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.194465    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-openshift-ca\") pod \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\" (UID: \"8a0f31fe-5f6f-435e-b794-7a80f39c76c5\") "
Oct 30 13:46:05.194857 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.194829    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-openshift-ca" pod "8a0f31fe-5f6f-435e-b794-7a80f39c76c5" (UID: "8a0f31fe-5f6f-435e-b794-7a80f39c76c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:05.195159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.195140    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:05.196737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.196705    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-kube-api-access-tlqz2" pod "8a0f31fe-5f6f-435e-b794-7a80f39c76c5" (UID: "8a0f31fe-5f6f-435e-b794-7a80f39c76c5"). InnerVolumeSpecName "kube-api-access-tlqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:05.197087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.197058    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-xks-ca" pod "8a0f31fe-5f6f-435e-b794-7a80f39c76c5" (UID: "8a0f31fe-5f6f-435e-b794-7a80f39c76c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:05.296888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.296785    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:05.296888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.296824    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlqz2\" (UniqueName: \"kubernetes.io/projected/8a0f31fe-5f6f-435e-b794-7a80f39c76c5-kube-api-access-tlqz2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:05.311667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.311624    4993 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:46:05.311880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:05.311690    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:46:06.000168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.000132    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"
Oct 30 13:46:06.000168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.000150    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt" event={"ID":"8a0f31fe-5f6f-435e-b794-7a80f39c76c5","Type":"ContainerDied","Data":"f66d1a0b132a0e16b390e22ed30efafcd92257ca9127d2cfb701fc14289b6156"}
Oct 30 13:46:06.000441 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.000287    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f66d1a0b132a0e16b390e22ed30efafcd92257ca9127d2cfb701fc14289b6156"
Oct 30 13:46:06.603917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.603890    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:46:06.716264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716223    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-envoy\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.716477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716314    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-kube-api-access-br7v4\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-kube-api-access-br7v4\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.716477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716357    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istiod-ca-cert\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.716613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716559    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-socket\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.716672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716636    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-certs\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.716743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716724    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-podinfo\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.716801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716781    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-credential-socket\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.716801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716791    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istiod-ca-cert" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:06.716915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716817    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-socket" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.716915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716822    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-token\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-token\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.716915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716894    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-data\") pod \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\" (UID: \"650859ec-868e-46f6-8a0a-1ab7c5ab14cc\") "
Oct 30 13:46:06.717099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.716935    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-certs" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.717169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.717090    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-credential-socket" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.717246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.717226    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-data" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.717717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.717691    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.717836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.717723    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.717836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.717778    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.717836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.717797    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.717836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.717810    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.720135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.720111    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-podinfo" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:06.720391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.720368    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-envoy" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.720478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.720411    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-token" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:06.720551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.720535    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-kube-api-access-br7v4" pod "650859ec-868e-46f6-8a0a-1ab7c5ab14cc" (UID: "650859ec-868e-46f6-8a0a-1ab7c5ab14cc"). InnerVolumeSpecName "kube-api-access-br7v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:06.819293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.819251    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.819293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.819287    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br7v4\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-kube-api-access-br7v4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.819293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.819299    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.819572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:06.819308    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650859ec-868e-46f6-8a0a-1ab7c5ab14cc-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:07.012862 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.012822    4993 generic.go:356] "Generic (PLEG): container finished" podID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerID="70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86" exitCode=0
Oct 30 13:46:07.013046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.012873    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" event={"ID":"650859ec-868e-46f6-8a0a-1ab7c5ab14cc","Type":"ContainerDied","Data":"70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86"}
Oct 30 13:46:07.013046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.012884    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"
Oct 30 13:46:07.013046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.012917    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6" event={"ID":"650859ec-868e-46f6-8a0a-1ab7c5ab14cc","Type":"ContainerDied","Data":"6bdaa0427180c7f8bf0f54509217a8f0e11b13eac937e525fe3125d750605185"}
Oct 30 13:46:07.013046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.012941    4993 scope.go:122] "RemoveContainer" containerID="70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86"
Oct 30 13:46:07.026806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.026783    4993 scope.go:122] "RemoveContainer" containerID="27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88"
Oct 30 13:46:07.042920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.042895    4993 scope.go:122] "RemoveContainer" containerID="70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86"
Oct 30 13:46:07.043707 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:07.043398    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86\": container with ID starting with 70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86 not found: ID does not exist" containerID="70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86"
Oct 30 13:46:07.043707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.043440    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86"} err="failed to get container status \"70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86\": rpc error: code = NotFound desc = could not find container \"70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86\": container with ID starting with 70e92dbf443900baa0a9a0498dd2728b2cfdad5826128f6b1456c30e311d3c86 not found: ID does not exist"
Oct 30 13:46:07.043707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.043462    4993 scope.go:122] "RemoveContainer" containerID="27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88"
Oct 30 13:46:07.043846 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:07.043788    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88\": container with ID starting with 27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88 not found: ID does not exist" containerID="27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88"
Oct 30 13:46:07.043846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.043806    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88"} err="failed to get container status \"27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88\": rpc error: code = NotFound desc = could not find container \"27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88\": container with ID starting with 27889d58549f8dd35a73d6487d2195a4cf61e3ae836b50609730c4d686a58b88 not found: ID does not exist"
Oct 30 13:46:07.094576 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.094540    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"]
Oct 30 13:46:07.138252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.138169    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-glgv6"]
Oct 30 13:46:07.694099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:07.694063    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" path="/var/lib/kubelet/pods/650859ec-868e-46f6-8a0a-1ab7c5ab14cc/volumes"
Oct 30 13:46:08.775014 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:08.774969    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:46:19.726382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:19.726336    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"]
Oct 30 13:46:19.727177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:19.727151    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b7407-6vd8t"]
Oct 30 13:46:21.694493 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:21.694456    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1faa809-72e9-49a2-8fa4-e97d9de1a166" path="/var/lib/kubelet/pods/a1faa809-72e9-49a2-8fa4-e97d9de1a166/volumes"
Oct 30 13:46:22.811737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:22.811694    4993 scope.go:122] "RemoveContainer" containerID="62160013bd807ef3f6f80c0673e77b55b7dae018bd2e157bcb216373c8d58bba"
Oct 30 13:46:27.283886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:27.283848    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"]
Oct 30 13:46:27.284749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:27.284511    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy" containerID="cri-o://071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d" gracePeriod=30
Oct 30 13:46:32.575817 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.575779    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:46:32.698914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.698851    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-certs\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.699305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.699273    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-token\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-token\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.699431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.699370    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-certs" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.699431 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.699409    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-kube-api-access-dnb5n\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-kube-api-access-dnb5n\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.699565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.699473    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-envoy\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.699565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.699521    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-socket\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.699732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.699704    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-data\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.699985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.699962    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-data" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.700081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.699987    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-podinfo\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.700148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.700116    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istiod-ca-cert\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.700224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.700182    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-credential-socket\") pod \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\" (UID: \"ed8cd26d-ec22-4f27-9565-26f6e066c1f8\") "
Oct 30 13:46:32.700347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.700322    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-socket" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.700714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.700693    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istiod-ca-cert" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:32.700951 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.700931    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-credential-socket" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.701069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.700933    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.701152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.701082    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.701152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.701100    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.701152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.701115    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.702006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.701979    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-token" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:32.702476 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.702449    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-kube-api-access-dnb5n" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "kube-api-access-dnb5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:32.703157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.703135    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-podinfo" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:32.703436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.703412    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-envoy" pod "ed8cd26d-ec22-4f27-9565-26f6e066c1f8" (UID: "ed8cd26d-ec22-4f27-9565-26f6e066c1f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.817879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.814516    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.817879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.814560    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.817879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.814577    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.817879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.814591    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnb5n\" (UniqueName: \"kubernetes.io/projected/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-kube-api-access-dnb5n\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.817879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:32.814604    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd26d-ec22-4f27-9565-26f6e066c1f8-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:33.292729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.292688    4993 generic.go:356] "Generic (PLEG): container finished" podID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerID="071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d" exitCode=0
Oct 30 13:46:33.292957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.292752    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"
Oct 30 13:46:33.292957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.292780    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" event={"ID":"ed8cd26d-ec22-4f27-9565-26f6e066c1f8","Type":"ContainerDied","Data":"071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d"}
Oct 30 13:46:33.292957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.292855    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz" event={"ID":"ed8cd26d-ec22-4f27-9565-26f6e066c1f8","Type":"ContainerDied","Data":"61a07264ea9a27083373793488ec7e4fa26ab95c435bb5e90c187da11d138da4"}
Oct 30 13:46:33.292957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.292882    4993 scope.go:122] "RemoveContainer" containerID="071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d"
Oct 30 13:46:33.307535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.307511    4993 scope.go:122] "RemoveContainer" containerID="d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7"
Oct 30 13:46:33.320049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.320024    4993 scope.go:122] "RemoveContainer" containerID="071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d"
Oct 30 13:46:33.320452 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:33.320428    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d\": container with ID starting with 071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d not found: ID does not exist" containerID="071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d"
Oct 30 13:46:33.320608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.320466    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d"} err="failed to get container status \"071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d\": rpc error: code = NotFound desc = could not find container \"071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d\": container with ID starting with 071e0c25fdcdcd5ea895e8602a1398edad130e1187ddaa5b8812c8a593b7ae5d not found: ID does not exist"
Oct 30 13:46:33.320608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.320492    4993 scope.go:122] "RemoveContainer" containerID="d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7"
Oct 30 13:46:33.321367 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:33.320833    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7\": container with ID starting with d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7 not found: ID does not exist" containerID="d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7"
Oct 30 13:46:33.321367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.320858    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7"} err="failed to get container status \"d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7\": rpc error: code = NotFound desc = could not find container \"d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7\": container with ID starting with d088aa8928108ae1e7072825235cb76194fdda8349fb98a6e4fed819319874e7 not found: ID does not exist"
Oct 30 13:46:33.365494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.365434    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"]
Oct 30 13:46:33.378911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.378867    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-l49nz"]
Oct 30 13:46:33.693277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:33.693162    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" path="/var/lib/kubelet/pods/ed8cd26d-ec22-4f27-9565-26f6e066c1f8/volumes"
Oct 30 13:46:38.570810 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:38.570767    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:46:40.046663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:40.046622    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"]
Oct 30 13:46:40.058341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:40.058286    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a93bd-twspv"]
Oct 30 13:46:40.585622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:40.585575    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"]
Oct 30 13:46:40.597481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:40.597438    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f97234-w8xp6"]
Oct 30 13:46:41.694770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:41.694724    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef7ff44-9b68-448a-825c-8e92cb0d7961" path="/var/lib/kubelet/pods/7ef7ff44-9b68-448a-825c-8e92cb0d7961/volumes"
Oct 30 13:46:41.695434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:41.695409    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b884ff9b-adfa-4c81-8282-09def8b358a8" path="/var/lib/kubelet/pods/b884ff9b-adfa-4c81-8282-09def8b358a8/volumes"
Oct 30 13:46:48.529953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:48.529906    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"]
Oct 30 13:46:48.534472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:48.534428    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" containerID="cri-o://8a7e09385749d2f4f65027bee9a288d5f9371bb9a7c2f19644cc19c38034d3ec" gracePeriod=30
Oct 30 13:46:49.136426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:49.136387    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"]
Oct 30 13:46:49.136923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:49.136885    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy" containerID="cri-o://5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2" gracePeriod=30
Oct 30 13:46:49.150881 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:49.150841    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"]
Oct 30 13:46:49.151376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:49.151338    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy" containerID="cri-o://66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00" gracePeriod=30
Oct 30 13:46:49.820231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:49.819298    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"]
Oct 30 13:46:49.829958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:49.823557    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af8369-q82jt"]
Oct 30 13:46:50.746745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.746710    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"]
Oct 30 13:46:50.747651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747619    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy"
Oct 30 13:46:50.747651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747648    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy"
Oct 30 13:46:50.747859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747666    4993 cpu_manager.go:414] "containerMap: removing container" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy"
Oct 30 13:46:50.747859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747674    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy"
Oct 30 13:46:50.747859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747702    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy"
Oct 30 13:46:50.747859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747710    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy"
Oct 30 13:46:50.747859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747722    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8a0f31fe-5f6f-435e-b794-7a80f39c76c5" containerName="revoke-keys"
Oct 30 13:46:50.747859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747728    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0f31fe-5f6f-435e-b794-7a80f39c76c5" containerName="revoke-keys"
Oct 30 13:46:50.747859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747738    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy"
Oct 30 13:46:50.747859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747745    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy"
Oct 30 13:46:50.748351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747942    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy"
Oct 30 13:46:50.748351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747958    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy"
Oct 30 13:46:50.748351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747967    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0f31fe-5f6f-435e-b794-7a80f39c76c5" containerName="revoke-keys"
Oct 30 13:46:50.748351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747975    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8cd26d-ec22-4f27-9565-26f6e066c1f8" containerName="istio-proxy"
Oct 30 13:46:50.748351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747984    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5eb4780-0ad4-49fb-aa2e-3ac4986c8d30" containerName="istio-proxy"
Oct 30 13:46:50.748351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.747998    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy"
Oct 30 13:46:50.755056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.755029    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.783253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.783166    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"]
Oct 30 13:46:50.785090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.784830    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d01cd5a5-openshift-default-dockercfg-rfjlf\""
Oct 30 13:46:50.839174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839134    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-certs\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.839625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839292    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-token\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.839625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839360    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-credential-socket\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.839625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839507    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-data\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.839625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839586    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-kube-api-access-bgnhw\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.839625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839615    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-envoy\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.839832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839637    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-socket\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.839832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839701    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istiod-ca-cert\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.839832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.839762    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-podinfo\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946286    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-data\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946395    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-kube-api-access-bgnhw\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946433    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-envoy\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946462    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-socket\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946506    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istiod-ca-cert\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946551    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-podinfo\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946617    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-certs\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946706    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-token\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.946736    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-credential-socket\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.947352    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-credential-socket\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.947635    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-socket\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.948565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.947871    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-data\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.949497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.948881    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istiod-ca-cert\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.964637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.964600    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-certs\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.972790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.972752    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-podinfo\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.980705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.980674    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-envoy\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:50.988961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.988927    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-kube-api-access-bgnhw\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:51.001369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:50.992779    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-token\") pod \"e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") " pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:51.100230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.097274    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:51.546962 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.546931    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"]
Oct 30 13:46:51.548868 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:46:51.548822    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6264a5b2_8654_48bc_a5e8_9d26c7e6acd6.slice/crio-0af9bee3864633400c3c2591560f426c66446ca3e36704d4ab7d6570c361b48f WatchSource:0}: Error finding container 0af9bee3864633400c3c2591560f426c66446ca3e36704d4ab7d6570c361b48f: Status 404 returned error can't find the container with id 0af9bee3864633400c3c2591560f426c66446ca3e36704d4ab7d6570c361b48f
Oct 30 13:46:51.552184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.552141    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:46:51.552310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.552290    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:46:51.552382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.552366    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:46:51.700256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.697891    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0f31fe-5f6f-435e-b794-7a80f39c76c5" path="/var/lib/kubelet/pods/8a0f31fe-5f6f-435e-b794-7a80f39c76c5/volumes"
Oct 30 13:46:51.876558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.876455    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" event={"ID":"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6","Type":"ContainerStarted","Data":"61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052"}
Oct 30 13:46:51.877161 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.877135    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" event={"ID":"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6","Type":"ContainerStarted","Data":"0af9bee3864633400c3c2591560f426c66446ca3e36704d4ab7d6570c361b48f"}
Oct 30 13:46:51.927743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:51.927680    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podStartSLOduration=1.927665382 podStartE2EDuration="1.927665382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:51.926284329 +0000 UTC m=+4476.402003939" watchObservedRunningTime="2026-10-30 13:46:51.927665382 +0000 UTC m=+4476.403385036"
Oct 30 13:46:52.103136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:52.100267    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:52.109382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:52.109340    4993 patch_prober.go:29] interesting pod/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Oct 30 13:46:52.109652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:52.109628    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Oct 30 13:46:53.104632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.104590    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:53.617721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.617686    4993 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:46:53.617913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.617751    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 13:46:53.899687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.899641    4993 generic.go:356] "Generic (PLEG): container finished" podID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerID="8a7e09385749d2f4f65027bee9a288d5f9371bb9a7c2f19644cc19c38034d3ec" exitCode=0
Oct 30 13:46:53.899896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.899735    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" event={"ID":"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3","Type":"ContainerDied","Data":"8a7e09385749d2f4f65027bee9a288d5f9371bb9a7c2f19644cc19c38034d3ec"}
Oct 30 13:46:53.899896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.899810    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n" event={"ID":"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3","Type":"ContainerDied","Data":"c36896160b2468637c1c4adbb76cccec870c133b3fbf60188e843332ec12e34b"}
Oct 30 13:46:53.899896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.899834    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c36896160b2468637c1c4adbb76cccec870c133b3fbf60188e843332ec12e34b"
Oct 30 13:46:53.899896 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.899861    4993 scope.go:122] "RemoveContainer" containerID="ac9f5c3d7811277e72cd240634adc79a58747b47ff25350ee37fede2f4a5cca1"
Oct 30 13:46:53.900484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.900465    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:53.901791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.901773    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:46:53.914871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:53.914845    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:46:54.021438 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021395    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-data\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.021637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021454    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istiod-ca-cert\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.021637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021493    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-podinfo\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.021637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021551    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-token\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-token\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.021637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021599    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-certs\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.021815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021722    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-data" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.021815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021741    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-socket\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.021900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021824    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-credential-socket\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.021944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.021931    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-envoy\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.022036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022010    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-certs" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.022036 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022025    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-socket" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.023026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022162    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-kube-api-access-tn7sd\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-kube-api-access-tn7sd\") pod \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\" (UID: \"c4d06911-a208-47f8-bad9-3cf4cf4e1ab3\") "
Oct 30 13:46:54.023026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022242    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-credential-socket" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.023026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022292    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istiod-ca-cert" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:54.023026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022915    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.023026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022937    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.023026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022953    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.023026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022967    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.023026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.022979    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.024286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.024186    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-token" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:54.024485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.024467    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-podinfo" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:54.024546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.024518    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-envoy" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.024835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.024819    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-kube-api-access-tn7sd" pod "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" (UID: "c4d06911-a208-47f8-bad9-3cf4cf4e1ab3"). InnerVolumeSpecName "kube-api-access-tn7sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:54.124987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.124935    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn7sd\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-kube-api-access-tn7sd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.124987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.124974    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.124987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.124990    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.125531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.125003    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.466980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.466946    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:46:54.506424 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.506387    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:46:54.531961 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.531921    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-certs\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.532148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532026    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-credential-socket\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.532148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532087    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-socket\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.532148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532128    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-data\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.532371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532191    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-istio-token\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-istio-token\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.532371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532230    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-kube-api-access-s874c\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-kube-api-access-s874c\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.532371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532285    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2710cdd-82a3-424e-938b-372e520fc95c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2710cdd-82a3-424e-938b-372e520fc95c-istiod-ca-cert\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.532371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532353    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2710cdd-82a3-424e-938b-372e520fc95c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2710cdd-82a3-424e-938b-372e520fc95c-istio-podinfo\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.532728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532706    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-data" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.532869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.532854    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-certs" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.533026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.533008    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-credential-socket" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.533139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.533114    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-socket" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.533289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.533263    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-envoy\") pod \"e2710cdd-82a3-424e-938b-372e520fc95c\" (UID: \"e2710cdd-82a3-424e-938b-372e520fc95c\") "
Oct 30 13:46:54.533382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.533365    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2710cdd-82a3-424e-938b-372e520fc95c-istiod-ca-cert" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:54.535282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.535175    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.535282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.535225    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.535282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.535242    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.535282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.535255    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.535282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.535268    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2710cdd-82a3-424e-938b-372e520fc95c-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.535764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.535740    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-envoy" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.536044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.536018    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-istio-token" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:54.536267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.536234    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2710cdd-82a3-424e-938b-372e520fc95c-istio-podinfo" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:54.536499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.536474    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-kube-api-access-s874c" pod "e2710cdd-82a3-424e-938b-372e520fc95c" (UID: "e2710cdd-82a3-424e-938b-372e520fc95c"). InnerVolumeSpecName "kube-api-access-s874c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636120    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-data\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636308    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-certs\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636361    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-socket\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636484    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-kube-api-access-8xq9g\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-kube-api-access-8xq9g\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636515    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-data" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636540    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-credential-socket\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636588    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-envoy\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636628    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72cc892-2538-41a7-8c6a-861a7775e4d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72cc892-2538-41a7-8c6a-861a7775e4d7-istiod-ca-cert\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636656    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-certs" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636679    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-token\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-token\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636696    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-podinfo\") pod \"a72cc892-2538-41a7-8c6a-861a7775e4d7\" (UID: \"a72cc892-2538-41a7-8c6a-861a7775e4d7\") "
Oct 30 13:46:54.637128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.636780    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-socket" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637282    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637305    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637320    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s874c\" (UniqueName: \"kubernetes.io/projected/e2710cdd-82a3-424e-938b-372e520fc95c-kube-api-access-s874c\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637336    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637351    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637364    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2710cdd-82a3-424e-938b-372e520fc95c-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637378    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2710cdd-82a3-424e-938b-372e520fc95c-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637486    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72cc892-2538-41a7-8c6a-861a7775e4d7-istiod-ca-cert" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:54.638079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.637887    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-credential-socket" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.640127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.640096    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-podinfo" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:54.641398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.641360    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-kube-api-access-8xq9g" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "kube-api-access-8xq9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:54.642570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.642546    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-envoy" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.642707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.642632    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-token" pod "a72cc892-2538-41a7-8c6a-861a7775e4d7" (UID: "a72cc892-2538-41a7-8c6a-861a7775e4d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:54.738883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.738839    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.738883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.738881    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72cc892-2538-41a7-8c6a-861a7775e4d7-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.739125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.738893    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.739125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.738905    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72cc892-2538-41a7-8c6a-861a7775e4d7-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.739125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.738920    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xq9g\" (UniqueName: \"kubernetes.io/projected/a72cc892-2538-41a7-8c6a-861a7775e4d7-kube-api-access-8xq9g\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.739125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.738933    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72cc892-2538-41a7-8c6a-861a7775e4d7-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.909834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.909799    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"
Oct 30 13:46:54.911448 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.911413    4993 generic.go:356] "Generic (PLEG): container finished" podID="e2710cdd-82a3-424e-938b-372e520fc95c" containerID="5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2" exitCode=0
Oct 30 13:46:54.911624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.911475    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"
Oct 30 13:46:54.911624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.911511    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" event={"ID":"e2710cdd-82a3-424e-938b-372e520fc95c","Type":"ContainerDied","Data":"5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2"}
Oct 30 13:46:54.911624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.911576    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7" event={"ID":"e2710cdd-82a3-424e-938b-372e520fc95c","Type":"ContainerDied","Data":"8380c61eb45aa3131b08c6fc34eaa02fd7e7785c710a0ef0b7d250103628ddaf"}
Oct 30 13:46:54.911624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.911608    4993 scope.go:122] "RemoveContainer" containerID="5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2"
Oct 30 13:46:54.918904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.916462    4993 generic.go:356] "Generic (PLEG): container finished" podID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerID="66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00" exitCode=0
Oct 30 13:46:54.918904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.916494    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" event={"ID":"a72cc892-2538-41a7-8c6a-861a7775e4d7","Type":"ContainerDied","Data":"66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00"}
Oct 30 13:46:54.918904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.916524    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"
Oct 30 13:46:54.918904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.916544    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc" event={"ID":"a72cc892-2538-41a7-8c6a-861a7775e4d7","Type":"ContainerDied","Data":"01bd36261861427d2d19b545ec6e753046dfb299e54d22b57176d4b0470ba952"}
Oct 30 13:46:54.932172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.932149    4993 scope.go:122] "RemoveContainer" containerID="5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2"
Oct 30 13:46:54.932551 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:54.932529    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2\": container with ID starting with 5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2 not found: ID does not exist" containerID="5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2"
Oct 30 13:46:54.932625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.932563    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2"} err="failed to get container status \"5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2\": rpc error: code = NotFound desc = could not find container \"5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2\": container with ID starting with 5f687e9cfe2fbe0340ea1f9908cf09b462e29041900a44a76239ad8618338bb2 not found: ID does not exist"
Oct 30 13:46:54.932625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.932579    4993 scope.go:122] "RemoveContainer" containerID="66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00"
Oct 30 13:46:54.944202 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.944168    4993 scope.go:122] "RemoveContainer" containerID="664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5"
Oct 30 13:46:54.956921 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.956891    4993 scope.go:122] "RemoveContainer" containerID="66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00"
Oct 30 13:46:54.957345 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:54.957318    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00\": container with ID starting with 66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00 not found: ID does not exist" containerID="66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00"
Oct 30 13:46:54.957453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.957353    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00"} err="failed to get container status \"66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00\": rpc error: code = NotFound desc = could not find container \"66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00\": container with ID starting with 66777bb432c14ba72e521e964344153ad1f3a0b78ebceeb9ae9ace7ec66aaa00 not found: ID does not exist"
Oct 30 13:46:54.957453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.957374    4993 scope.go:122] "RemoveContainer" containerID="664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5"
Oct 30 13:46:54.957710 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:46:54.957690    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5\": container with ID starting with 664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5 not found: ID does not exist" containerID="664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5"
Oct 30 13:46:54.957776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:54.957716    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5"} err="failed to get container status \"664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5\": rpc error: code = NotFound desc = could not find container \"664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5\": container with ID starting with 664eb189b7808b437a91708793b1b45fb0973c6a21924912f83d9f98d4c7b4f5 not found: ID does not exist"
Oct 30 13:46:55.047876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.047786    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"]
Oct 30 13:46:55.054972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.054939    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-ql2fc"]
Oct 30 13:46:55.103282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.103244    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"]
Oct 30 13:46:55.110261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.110220    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-t6kd7"]
Oct 30 13:46:55.145086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.145048    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"]
Oct 30 13:46:55.156916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.156876    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-2vj2n"]
Oct 30 13:46:55.695056 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.695017    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" path="/var/lib/kubelet/pods/a72cc892-2538-41a7-8c6a-861a7775e4d7/volumes"
Oct 30 13:46:55.695633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.695616    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" path="/var/lib/kubelet/pods/c4d06911-a208-47f8-bad9-3cf4cf4e1ab3/volumes"
Oct 30 13:46:55.696107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:55.696092    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" path="/var/lib/kubelet/pods/e2710cdd-82a3-424e-938b-372e520fc95c/volumes"
Oct 30 13:46:56.367385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.367340    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"]
Oct 30 13:46:56.367959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.367942    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.367959    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.367979    4993 cpu_manager.go:414] "containerMap: removing container" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.367985    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650859ec-868e-46f6-8a0a-1ab7c5ab14cc" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.367995    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368001    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368010    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368017    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368032    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy"
Oct 30 13:46:56.368051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368037    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy"
Oct 30 13:46:56.368597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368134    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy"
Oct 30 13:46:56.368597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368147    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy"
Oct 30 13:46:56.368597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368156    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy"
Oct 30 13:46:56.368597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368163    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2710cdd-82a3-424e-938b-372e520fc95c" containerName="istio-proxy"
Oct 30 13:46:56.368597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.368170    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d06911-a208-47f8-bad9-3cf4cf4e1ab3" containerName="istio-proxy"
Oct 30 13:46:56.372925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.372896    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:56.388033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.387998    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d01cd5a5-dockercfg-n62dw\""
Oct 30 13:46:56.400699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.400660    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d01cd5a5\""
Oct 30 13:46:56.407255 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.407190    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"]
Oct 30 13:46:56.539231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.538469    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af20f1b-1631-484d-8f4d-6dc472486867-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp\" (UID: \"2af20f1b-1631-484d-8f4d-6dc472486867\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:56.539231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.538653    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvtg\" (UniqueName: \"kubernetes.io/projected/2af20f1b-1631-484d-8f4d-6dc472486867-kube-api-access-lcvtg\") pod \"payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp\" (UID: \"2af20f1b-1631-484d-8f4d-6dc472486867\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:56.694342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.671303    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvtg\" (UniqueName: \"kubernetes.io/projected/2af20f1b-1631-484d-8f4d-6dc472486867-kube-api-access-lcvtg\") pod \"payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp\" (UID: \"2af20f1b-1631-484d-8f4d-6dc472486867\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:56.694342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.671598    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af20f1b-1631-484d-8f4d-6dc472486867-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp\" (UID: \"2af20f1b-1631-484d-8f4d-6dc472486867\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:56.694342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.672508    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af20f1b-1631-484d-8f4d-6dc472486867-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp\" (UID: \"2af20f1b-1631-484d-8f4d-6dc472486867\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:56.721663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.716441    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvtg\" (UniqueName: \"kubernetes.io/projected/2af20f1b-1631-484d-8f4d-6dc472486867-kube-api-access-lcvtg\") pod \"payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp\" (UID: \"2af20f1b-1631-484d-8f4d-6dc472486867\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:56.992977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:56.992939    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:57.310093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:57.310056    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"]
Oct 30 13:46:57.310637 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:46:57.310611    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af20f1b_1631_484d_8f4d_6dc472486867.slice/crio-6185c0f3bfa9a02acaa6d527fb24bbb408fb263ac971f16c1ecd83cd1c81adee WatchSource:0}: Error finding container 6185c0f3bfa9a02acaa6d527fb24bbb408fb263ac971f16c1ecd83cd1c81adee: Status 404 returned error can't find the container with id 6185c0f3bfa9a02acaa6d527fb24bbb408fb263ac971f16c1ecd83cd1c81adee
Oct 30 13:46:57.946408 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:57.946369    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp" event={"ID":"2af20f1b-1631-484d-8f4d-6dc472486867","Type":"ContainerStarted","Data":"403d20828c15358435fefb84b1d1596b098a8fda4292e55c94ab1b5f278c74b5"}
Oct 30 13:46:57.946810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:57.946430    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:46:57.946810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:57.946454    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp" event={"ID":"2af20f1b-1631-484d-8f4d-6dc472486867","Type":"ContainerStarted","Data":"6185c0f3bfa9a02acaa6d527fb24bbb408fb263ac971f16c1ecd83cd1c81adee"}
Oct 30 13:46:58.001353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:58.001279    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp" podStartSLOduration=2.001254231 podStartE2EDuration="2.001254231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:57.99470778 +0000 UTC m=+4482.470427390" watchObservedRunningTime="2026-10-30 13:46:58.001254231 +0000 UTC m=+4482.476973861"
Oct 30 13:46:58.025230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:58.022514    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"]
Oct 30 13:46:58.025230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:46:58.022743    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy" containerID="cri-o://8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca" gracePeriod=30
Oct 30 13:47:02.445318 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.445278    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"]
Oct 30 13:47:02.445873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.445855    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy"
Oct 30 13:47:02.445918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.445874    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72cc892-2538-41a7-8c6a-861a7775e4d7" containerName="istio-proxy"
Oct 30 13:47:02.450029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.450004    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.465630 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.465599    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:47:02.470871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.470797    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"]
Oct 30 13:47:02.477789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.477751    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:47:02.477789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.477770    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:47:02.478141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.478113    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:47:02.587947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.587894    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbg9\" (UniqueName: \"kubernetes.io/projected/786c1728-4779-43d9-b7a6-b13c50639968-kube-api-access-rrbg9\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.588155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.587963    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.588155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.588093    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.690064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.690020    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.690280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.690129    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.690469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.690406    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbg9\" (UniqueName: \"kubernetes.io/projected/786c1728-4779-43d9-b7a6-b13c50639968-kube-api-access-rrbg9\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.690925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.690856    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.694179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.694141    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.714878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.714803    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbg9\" (UniqueName: \"kubernetes.io/projected/786c1728-4779-43d9-b7a6-b13c50639968-kube-api-access-rrbg9\") pod \"maas-api-revoke-sub-c495032a-rlccd\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:02.765915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:02.765872    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:03.088187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.088151    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"]
Oct 30 13:47:03.122795 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:47:03.122759    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786c1728_4779_43d9_b7a6_b13c50639968.slice/crio-5902bfff95bd0150f71b241a2de354e2dc59e1baf466b8aff147e2c72ee88059 WatchSource:0}: Error finding container 5902bfff95bd0150f71b241a2de354e2dc59e1baf466b8aff147e2c72ee88059: Status 404 returned error can't find the container with id 5902bfff95bd0150f71b241a2de354e2dc59e1baf466b8aff147e2c72ee88059
Oct 30 13:47:03.377762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.377722    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:47:03.506031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.505977    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-socket\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.506656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.506579    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-data\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.506752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.506685    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-credential-socket\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.506828 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.506788    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-envoy\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.506897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.506864    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-kube-api-access-876x7\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-kube-api-access-876x7\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.506970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.506918    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d403bfbb-dd29-451b-9477-5ffae3262362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403bfbb-dd29-451b-9477-5ffae3262362-istio-podinfo\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.506970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.506954    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-istio-token\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-istio-token\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.506970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.506949    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-socket" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:47:03.507147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507030    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d403bfbb-dd29-451b-9477-5ffae3262362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403bfbb-dd29-451b-9477-5ffae3262362-istiod-ca-cert\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.507147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507072    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-certs\") pod \"d403bfbb-dd29-451b-9477-5ffae3262362\" (UID: \"d403bfbb-dd29-451b-9477-5ffae3262362\") "
Oct 30 13:47:03.507292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507149    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-data" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:47:03.507503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507449    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-credential-socket" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:47:03.508068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507784    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:03.508068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507809    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:03.508068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507832    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:03.508068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507895    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d403bfbb-dd29-451b-9477-5ffae3262362-istiod-ca-cert" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:03.508068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.507976    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-certs" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:47:03.509722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.509695    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-istio-token" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:03.509854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.509837    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-envoy" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:47:03.509987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.509960    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d403bfbb-dd29-451b-9477-5ffae3262362-istio-podinfo" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:47:03.510185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.510157    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-kube-api-access-876x7" pod "d403bfbb-dd29-451b-9477-5ffae3262362" (UID: "d403bfbb-dd29-451b-9477-5ffae3262362"). InnerVolumeSpecName "kube-api-access-876x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:03.609733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.609629    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d403bfbb-dd29-451b-9477-5ffae3262362-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:03.609733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.609678    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:03.609733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.609692    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d403bfbb-dd29-451b-9477-5ffae3262362-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:03.609733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.609709    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-876x7\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-kube-api-access-876x7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:03.609733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.609724    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d403bfbb-dd29-451b-9477-5ffae3262362-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:03.609733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:03.609738    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d403bfbb-dd29-451b-9477-5ffae3262362-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:04.010163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.010111    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd" event={"ID":"786c1728-4779-43d9-b7a6-b13c50639968","Type":"ContainerStarted","Data":"5129916921da61b51274e2c627a5850373afb5c97a29708f9c79aee96d76630b"}
Oct 30 13:47:04.010401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.010239    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd" event={"ID":"786c1728-4779-43d9-b7a6-b13c50639968","Type":"ContainerStarted","Data":"5902bfff95bd0150f71b241a2de354e2dc59e1baf466b8aff147e2c72ee88059"}
Oct 30 13:47:04.012194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.012162    4993 generic.go:356] "Generic (PLEG): container finished" podID="d403bfbb-dd29-451b-9477-5ffae3262362" containerID="8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca" exitCode=0
Oct 30 13:47:04.012349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.012241    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"
Oct 30 13:47:04.012349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.012237    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" event={"ID":"d403bfbb-dd29-451b-9477-5ffae3262362","Type":"ContainerDied","Data":"8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca"}
Oct 30 13:47:04.012425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.012364    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8" event={"ID":"d403bfbb-dd29-451b-9477-5ffae3262362","Type":"ContainerDied","Data":"66447a1f60cfd479b82f6cf5f5336fc95d7e89401607ce0c25932f56eeef422b"}
Oct 30 13:47:04.012425 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.012394    4993 scope.go:122] "RemoveContainer" containerID="8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca"
Oct 30 13:47:04.025873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.025847    4993 scope.go:122] "RemoveContainer" containerID="e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281"
Oct 30 13:47:04.038398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.038374    4993 scope.go:122] "RemoveContainer" containerID="8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca"
Oct 30 13:47:04.038761 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:47:04.038735    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca\": container with ID starting with 8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca not found: ID does not exist" containerID="8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca"
Oct 30 13:47:04.038858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.038774    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca"} err="failed to get container status \"8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca\": rpc error: code = NotFound desc = could not find container \"8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca\": container with ID starting with 8e6d6dbc244377d013f737fe9f7524b6e595e4eeb69636175f9bc3de5e7529ca not found: ID does not exist"
Oct 30 13:47:04.038858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.038793    4993 scope.go:122] "RemoveContainer" containerID="e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281"
Oct 30 13:47:04.039088 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:47:04.039072    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281\": container with ID starting with e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281 not found: ID does not exist" containerID="e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281"
Oct 30 13:47:04.039137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.039093    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281"} err="failed to get container status \"e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281\": rpc error: code = NotFound desc = could not find container \"e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281\": container with ID starting with e4957369268acc0cfbfb3063d3d226992f2c67d95fde3e961fd7a353b8acf281 not found: ID does not exist"
Oct 30 13:47:04.058182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.058113    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd" podStartSLOduration=2.058092214 podStartE2EDuration="2.058092214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:04.05308978 +0000 UTC m=+4488.528809449" watchObservedRunningTime="2026-10-30 13:47:04.058092214 +0000 UTC m=+4488.533811827"
Oct 30 13:47:04.092468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.092426    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"]
Oct 30 13:47:04.104462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:04.104417    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-af8369-openshift-default-6559ddb96c-pc9m8"]
Oct 30 13:47:05.022003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:05.021959    4993 generic.go:356] "Generic (PLEG): container finished" podID="786c1728-4779-43d9-b7a6-b13c50639968" containerID="5129916921da61b51274e2c627a5850373afb5c97a29708f9c79aee96d76630b" exitCode=0
Oct 30 13:47:05.022694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:05.022150    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd" event={"ID":"786c1728-4779-43d9-b7a6-b13c50639968","Type":"ContainerDied","Data":"5129916921da61b51274e2c627a5850373afb5c97a29708f9c79aee96d76630b"}
Oct 30 13:47:05.692731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:05.692697    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" path="/var/lib/kubelet/pods/d403bfbb-dd29-451b-9477-5ffae3262362/volumes"
Oct 30 13:47:06.229495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.229461    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:06.243789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.243736    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-openshift-ca\") pod \"786c1728-4779-43d9-b7a6-b13c50639968\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") "
Oct 30 13:47:06.244051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.243966    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786c1728-4779-43d9-b7a6-b13c50639968-kube-api-access-rrbg9\" (UniqueName: \"kubernetes.io/projected/786c1728-4779-43d9-b7a6-b13c50639968-kube-api-access-rrbg9\") pod \"786c1728-4779-43d9-b7a6-b13c50639968\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") "
Oct 30 13:47:06.244051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.244039    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-xks-ca\") pod \"786c1728-4779-43d9-b7a6-b13c50639968\" (UID: \"786c1728-4779-43d9-b7a6-b13c50639968\") "
Oct 30 13:47:06.245534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.245501    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-openshift-ca" pod "786c1728-4779-43d9-b7a6-b13c50639968" (UID: "786c1728-4779-43d9-b7a6-b13c50639968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:06.247244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.247180    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-xks-ca" pod "786c1728-4779-43d9-b7a6-b13c50639968" (UID: "786c1728-4779-43d9-b7a6-b13c50639968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:47:06.247597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.247550    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786c1728-4779-43d9-b7a6-b13c50639968-kube-api-access-rrbg9" pod "786c1728-4779-43d9-b7a6-b13c50639968" (UID: "786c1728-4779-43d9-b7a6-b13c50639968"). InnerVolumeSpecName "kube-api-access-rrbg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:06.345814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.345705    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:06.345814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.345747    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrbg9\" (UniqueName: \"kubernetes.io/projected/786c1728-4779-43d9-b7a6-b13c50639968-kube-api-access-rrbg9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:06.345814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:06.345759    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786c1728-4779-43d9-b7a6-b13c50639968-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:47:07.040871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:07.040836    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"
Oct 30 13:47:07.040871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:07.040863    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd" event={"ID":"786c1728-4779-43d9-b7a6-b13c50639968","Type":"ContainerDied","Data":"5902bfff95bd0150f71b241a2de354e2dc59e1baf466b8aff147e2c72ee88059"}
Oct 30 13:47:07.041115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:07.040919    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5902bfff95bd0150f71b241a2de354e2dc59e1baf466b8aff147e2c72ee88059"
Oct 30 13:47:08.572720 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:47:08.572673    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:47:08.960163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:08.960065    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:47:16.538750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:16.538716    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:47:16.581449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:16.581414    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:47:22.075355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:22.075299    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:47:22.353291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:22.352345    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:47:22.801322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:22.791247    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:47:22.801861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:22.801817    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 13:47:23.051437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:23.051330    4993 scope.go:122] "RemoveContainer" containerID="76171f413860bd56d93d7416841126bfea385974dc6cd47eed0540620f12b0eb"
Oct 30 13:47:23.072626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:23.072599    4993 scope.go:122] "RemoveContainer" containerID="435f58a114f2c753c989b717e4a26223d21b214dedbe11890c903ae8a30c4527"
Oct 30 13:47:23.092226 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:23.092114    4993 scope.go:122] "RemoveContainer" containerID="8a7e09385749d2f4f65027bee9a288d5f9371bb9a7c2f19644cc19c38034d3ec"
Oct 30 13:47:23.167967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:23.167925    4993 scope.go:122] "RemoveContainer" containerID="cc8dc70876c2e3b8056e631c39e5ed6fbb5acdb78507775c9db23cde96ae4455"
Oct 30 13:47:38.586016 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:47:38.585966    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:47:40.139228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:40.136289    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 13:47:40.226286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:40.217340    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:47:40.671578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:40.671512    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:47:41.093230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:41.092549    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:47:41.101231 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:47:41.093903    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:48:08.076449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:08.076404    4993 generic.go:356] "Generic (PLEG): container finished" podID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerID="61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052" exitCode=0
Oct 30 13:48:08.077022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:08.076468    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" event={"ID":"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6","Type":"ContainerDied","Data":"61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052"}
Oct 30 13:48:08.077022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:08.076988    4993 scope.go:122] "RemoveContainer" containerID="61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052"
Oct 30 13:48:08.079999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:08.079953    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:48:08.080131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:08.080038    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:48:08.080131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:08.080072    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:48:08.572904 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:48:08.572856    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:48:08.900274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:08.900140    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:48:09.097159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:09.097123    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" event={"ID":"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6","Type":"ContainerStarted","Data":"88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301"}
Oct 30 13:48:09.099140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:09.099113    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:48:09.100687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:09.100655    4993 patch_prober.go:29] interesting pod/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Oct 30 13:48:09.100944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:09.100911    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Oct 30 13:48:10.099102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:10.099050    4993 patch_prober.go:29] interesting pod/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Oct 30 13:48:10.100004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:10.099113    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Oct 30 13:48:11.126230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:11.125690    4993 patch_prober.go:29] interesting pod/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Oct 30 13:48:11.126230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:11.125792    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Oct 30 13:48:12.099359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:12.099322    4993 patch_prober.go:29] interesting pod/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Oct 30 13:48:12.099566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:12.099388    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Oct 30 13:48:13.099341 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:13.099304    4993 patch_prober.go:29] interesting pod/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Oct 30 13:48:13.099789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:13.099370    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Oct 30 13:48:14.099119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:14.099076    4993 patch_prober.go:29] interesting pod/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Oct 30 13:48:14.099478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:14.099446    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Oct 30 13:48:15.104766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:15.104727    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:48:15.105344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:15.105222    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:48:15.106575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:15.106544    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:48:23.266393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.266350    4993 scope.go:122] "RemoveContainer" containerID="3dd569881fbe5d65ee1c4eaf11e97891ec365030cddbab90b2897d701e9521bd"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.738757    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"]
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739602    4993 cpu_manager.go:414] "containerMap: removing container" podUID="786c1728-4779-43d9-b7a6-b13c50639968" containerName="revoke-keys"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739620    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786c1728-4779-43d9-b7a6-b13c50639968" containerName="revoke-keys"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739652    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739659    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739687    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739696    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739853    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="786c1728-4779-43d9-b7a6-b13c50639968" containerName="revoke-keys"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739870    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy"
Oct 30 13:48:23.743656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.739880    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d403bfbb-dd29-451b-9477-5ffae3262362" containerName="istio-proxy"
Oct 30 13:48:23.744354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.743731    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.773877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.773843    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7a8b38\"/\"kube-root-ca.crt\""
Oct 30 13:48:23.774829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.774799    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-home\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.774931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.774852    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tls-certs\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.774931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.774916    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-dshm\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.775018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.774943    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-model-cache\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.775066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.775050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s88v\" (UniqueName: \"kubernetes.io/projected/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-kube-api-access-5s88v\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.775112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.775089    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tmp-dir\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.775587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.775560    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7a8b38\"/\"sub-test-model-7a8b38-kserve-self-signed-certs\""
Oct 30 13:48:23.775728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.775670    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7a8b38\"/\"openshift-service-ca.crt\""
Oct 30 13:48:23.776507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.775764    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7a8b38\"/\"default-dockercfg-7dnrc\""
Oct 30 13:48:23.778849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.778820    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"]
Oct 30 13:48:23.876416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.876365    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s88v\" (UniqueName: \"kubernetes.io/projected/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-kube-api-access-5s88v\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.876660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.876527    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tmp-dir\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.876751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.876685    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-home\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.876751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.876719    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tls-certs\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.876869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.876778    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-dshm\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.876869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.876806    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-model-cache\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.877080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.877006    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tmp-dir\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.877155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.877083    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-home\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.877604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.877568    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-model-cache\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.880374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.880316    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-dshm\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.881073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.881043    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tls-certs\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:23.904713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:23.904678    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s88v\" (UniqueName: \"kubernetes.io/projected/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-kube-api-access-5s88v\") pod \"sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:24.082282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:24.082007    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:24.497468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:24.497433    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"]
Oct 30 13:48:24.497910 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:48:24.497588    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3ed3ca_445a_4a0b_ae08_a136c26c55de.slice/crio-6d210e9f9bfc8333ff857fbc165da0ebb70bd4ef3064944a0d9ec30602611162 WatchSource:0}: Error finding container 6d210e9f9bfc8333ff857fbc165da0ebb70bd4ef3064944a0d9ec30602611162: Status 404 returned error can't find the container with id 6d210e9f9bfc8333ff857fbc165da0ebb70bd4ef3064944a0d9ec30602611162
Oct 30 13:48:25.308816 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:25.308770    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq" event={"ID":"7b3ed3ca-445a-4a0b-ae08-a136c26c55de","Type":"ContainerStarted","Data":"0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28"}
Oct 30 13:48:25.309051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:25.308829    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq" event={"ID":"7b3ed3ca-445a-4a0b-ae08-a136c26c55de","Type":"ContainerStarted","Data":"6d210e9f9bfc8333ff857fbc165da0ebb70bd4ef3064944a0d9ec30602611162"}
Oct 30 13:48:25.355899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:25.355822    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq" podStartSLOduration=2.355802751 podStartE2EDuration="2.355802751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:25.351266022 +0000 UTC m=+4569.826985633" watchObservedRunningTime="2026-10-30 13:48:25.355802751 +0000 UTC m=+4569.831522362"
Oct 30 13:48:26.084766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:26.084724    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:34.084030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:34.083986    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:34.097651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:34.097614    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:34.468344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:34.468246    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:48:38.570252 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:48:38.570194    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:48:43.643348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.643307    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"]
Oct 30 13:48:43.647480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.647451    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.663314 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.663280    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:48:43.677257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.677187    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:48:43.677847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.677824    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:48:43.702018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.701978    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"]
Oct 30 13:48:43.702366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.702332    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.702503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.702389    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.702549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.702525    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ht4\" (UniqueName: \"kubernetes.io/projected/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-kube-api-access-v8ht4\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.769906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.769865    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:48:43.803985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.803937    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.803985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.803989    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.804279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.804042    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ht4\" (UniqueName: \"kubernetes.io/projected/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-kube-api-access-v8ht4\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.804834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.804804    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.806819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.806790    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.829805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.829771    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ht4\" (UniqueName: \"kubernetes.io/projected/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-kube-api-access-v8ht4\") pod \"maas-api-revoke-sub-480237d9-qhpzz\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:43.965678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:43.965575    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:44.289868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:44.289821    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"]
Oct 30 13:48:44.545882 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:44.545790    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz" event={"ID":"0392f69a-8db8-4cc4-8202-fe06d5d1cd08","Type":"ContainerStarted","Data":"ed691e11633752299d753ad9444688dae76f562195553583ec4f9e42a23596a9"}
Oct 30 13:48:45.554988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:45.554945    4993 generic.go:356] "Generic (PLEG): container finished" podID="0392f69a-8db8-4cc4-8202-fe06d5d1cd08" containerID="0e3d3f84f12699f97d6d597418b54c57163ceb8b25f1239f1cae24c558b0c060" exitCode=0
Oct 30 13:48:45.555469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:45.555035    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz" event={"ID":"0392f69a-8db8-4cc4-8202-fe06d5d1cd08","Type":"ContainerDied","Data":"0e3d3f84f12699f97d6d597418b54c57163ceb8b25f1239f1cae24c558b0c060"}
Oct 30 13:48:46.807224 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.807182    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:46.845692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.845594    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-openshift-ca\") pod \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") "
Oct 30 13:48:46.845692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.845665    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-xks-ca\") pod \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") "
Oct 30 13:48:46.845930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.845777    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-kube-api-access-v8ht4\" (UniqueName: \"kubernetes.io/projected/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-kube-api-access-v8ht4\") pod \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\" (UID: \"0392f69a-8db8-4cc4-8202-fe06d5d1cd08\") "
Oct 30 13:48:46.845999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.845978    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-openshift-ca" pod "0392f69a-8db8-4cc4-8202-fe06d5d1cd08" (UID: "0392f69a-8db8-4cc4-8202-fe06d5d1cd08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:46.846351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.846327    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:48:46.847965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.847939    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-xks-ca" pod "0392f69a-8db8-4cc4-8202-fe06d5d1cd08" (UID: "0392f69a-8db8-4cc4-8202-fe06d5d1cd08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:48:46.848081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.848060    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-kube-api-access-v8ht4" pod "0392f69a-8db8-4cc4-8202-fe06d5d1cd08" (UID: "0392f69a-8db8-4cc4-8202-fe06d5d1cd08"). InnerVolumeSpecName "kube-api-access-v8ht4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:46.947845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.947798    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:48:46.947845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:46.947840    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ht4\" (UniqueName: \"kubernetes.io/projected/0392f69a-8db8-4cc4-8202-fe06d5d1cd08-kube-api-access-v8ht4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:48:47.574495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:47.574457    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"
Oct 30 13:48:47.574685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:47.574491    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz" event={"ID":"0392f69a-8db8-4cc4-8202-fe06d5d1cd08","Type":"ContainerDied","Data":"ed691e11633752299d753ad9444688dae76f562195553583ec4f9e42a23596a9"}
Oct 30 13:48:47.574685 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:47.574548    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed691e11633752299d753ad9444688dae76f562195553583ec4f9e42a23596a9"
Oct 30 13:48:50.780000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.779963    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"]
Oct 30 13:48:50.783906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.781078    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0392f69a-8db8-4cc4-8202-fe06d5d1cd08" containerName="revoke-keys"
Oct 30 13:48:50.783906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.781106    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0392f69a-8db8-4cc4-8202-fe06d5d1cd08" containerName="revoke-keys"
Oct 30 13:48:50.783906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.781341    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0392f69a-8db8-4cc4-8202-fe06d5d1cd08" containerName="revoke-keys"
Oct 30 13:48:50.785565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.785537    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:50.817061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.817023    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-e9cb4a\"/\"openshift-service-ca.crt\""
Oct 30 13:48:50.817279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.817092    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-e9cb4a\"/\"kube-root-ca.crt\""
Oct 30 13:48:50.817279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.817092    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-e9cb4a\"/\"sub-test-model-e9cb4a-kserve-self-signed-certs\""
Oct 30 13:48:50.817279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.817138    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-e9cb4a\"/\"default-dockercfg-2ktrv\""
Oct 30 13:48:50.821244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.821181    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"]
Oct 30 13:48:50.909460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.909409    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc29550-dc74-4259-8bba-5d6115bfb864-tls-certs\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:50.909460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.909466    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-model-cache\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:50.909773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.909555    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-dshm\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:50.909773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.909612    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqpp\" (UniqueName: \"kubernetes.io/projected/cbc29550-dc74-4259-8bba-5d6115bfb864-kube-api-access-tsqpp\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:50.909773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.909709    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-tmp-dir\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:50.909773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:50.909763    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-home\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.010652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.010608    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-dshm\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.010883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.010668    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqpp\" (UniqueName: \"kubernetes.io/projected/cbc29550-dc74-4259-8bba-5d6115bfb864-kube-api-access-tsqpp\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.010883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.010751    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-tmp-dir\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.010883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.010825    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-home\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.011082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.010973    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc29550-dc74-4259-8bba-5d6115bfb864-tls-certs\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.011082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.011013    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-model-cache\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.011250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.011183    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-tmp-dir\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.011339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.011296    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-home\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.011397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.011376    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-model-cache\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.013252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.013224    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-dshm\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.013574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.013552    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc29550-dc74-4259-8bba-5d6115bfb864-tls-certs\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.048109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.047975    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqpp\" (UniqueName: \"kubernetes.io/projected/cbc29550-dc74-4259-8bba-5d6115bfb864-kube-api-access-tsqpp\") pod \"sub-test-model-e9cb4a-kserve-565889df4f-z76tm\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") " pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.105570 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.105281    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:51.499766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.499730    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"]
Oct 30 13:48:51.501332 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:48:51.501301    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbc29550_dc74_4259_8bba_5d6115bfb864.slice/crio-819b370022609b607d4f70215c9920af4e5d0c02e1c5131b64a285ab72a06adc WatchSource:0}: Error finding container 819b370022609b607d4f70215c9920af4e5d0c02e1c5131b64a285ab72a06adc: Status 404 returned error can't find the container with id 819b370022609b607d4f70215c9920af4e5d0c02e1c5131b64a285ab72a06adc
Oct 30 13:48:51.618670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.618618    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm" event={"ID":"cbc29550-dc74-4259-8bba-5d6115bfb864","Type":"ContainerStarted","Data":"e448ffa20a7ecd83211f7cda1c9505b82753122c85b078ad72e6f467e17bedc2"}
Oct 30 13:48:51.618895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.618681    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm" event={"ID":"cbc29550-dc74-4259-8bba-5d6115bfb864","Type":"ContainerStarted","Data":"819b370022609b607d4f70215c9920af4e5d0c02e1c5131b64a285ab72a06adc"}
Oct 30 13:48:51.673382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:51.673310    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm" podStartSLOduration=1.6732886599999999 podStartE2EDuration="1.67328866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:51.670618292 +0000 UTC m=+4596.146337903" watchObservedRunningTime="2026-10-30 13:48:51.67328866 +0000 UTC m=+4596.149008272"
Oct 30 13:48:52.105973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:52.105934    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:48:55.058879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:55.058827    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"]
Oct 30 13:48:55.063239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:55.063181    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12f2c996-jpjzm"]
Oct 30 13:48:55.693531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:55.693493    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19405398-30b1-45b7-a28e-54c223cf12c5" path="/var/lib/kubelet/pods/19405398-30b1-45b7-a28e-54c223cf12c5/volumes"
Oct 30 13:48:56.945988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:56.945929    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"]
Oct 30 13:48:56.964912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:56.964879    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:56.981400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:56.981364    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"]
Oct 30 13:48:56.990303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:56.990275    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-98dabb\"/\"kube-root-ca.crt\""
Oct 30 13:48:57.002136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.002101    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-98dabb\"/\"default-dockercfg-gcxk7\""
Oct 30 13:48:57.002374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.002162    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-98dabb\"/\"openshift-service-ca.crt\""
Oct 30 13:48:57.002452 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.002397    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-98dabb\"/\"sub-test-model-98dabb-kserve-self-signed-certs\""
Oct 30 13:48:57.057651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.057609    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-tmp-dir\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.057848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.057678    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb93120-9646-4149-8c6d-01f6b52a2c37-tls-certs\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.057848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.057812    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxt7b\" (UniqueName: \"kubernetes.io/projected/deb93120-9646-4149-8c6d-01f6b52a2c37-kube-api-access-zxt7b\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.058077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.058056    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-dshm\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.058240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.058181    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-home\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.058363 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.058342    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-model-cache\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.160638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.160593    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-model-cache\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.160869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.160682    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-tmp-dir\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.160869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.160731    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb93120-9646-4149-8c6d-01f6b52a2c37-tls-certs\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.160869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.160758    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxt7b\" (UniqueName: \"kubernetes.io/projected/deb93120-9646-4149-8c6d-01f6b52a2c37-kube-api-access-zxt7b\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.161060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.160964    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-dshm\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.161118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.161056    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-home\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.161118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.161081    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-model-cache\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.161118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.161096    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-tmp-dir\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.161577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.161554    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-home\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.163812 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.163787    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-dshm\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.164706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.164677    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb93120-9646-4149-8c6d-01f6b52a2c37-tls-certs\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.214195 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.214009    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxt7b\" (UniqueName: \"kubernetes.io/projected/deb93120-9646-4149-8c6d-01f6b52a2c37-kube-api-access-zxt7b\") pod \"sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.286434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.286383    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:57.801123 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:48:57.801080    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeb93120_9646_4149_8c6d_01f6b52a2c37.slice/crio-824a40d9d4acd0de8558b3ed369788999758711f00265b6c5c9b028e0c2e5c95 WatchSource:0}: Error finding container 824a40d9d4acd0de8558b3ed369788999758711f00265b6c5c9b028e0c2e5c95: Status 404 returned error can't find the container with id 824a40d9d4acd0de8558b3ed369788999758711f00265b6c5c9b028e0c2e5c95
Oct 30 13:48:57.802595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:57.802557    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"]
Oct 30 13:48:58.681437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:58.681397    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz" event={"ID":"deb93120-9646-4149-8c6d-01f6b52a2c37","Type":"ContainerStarted","Data":"6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f"}
Oct 30 13:48:58.681437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:58.681446    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz" event={"ID":"deb93120-9646-4149-8c6d-01f6b52a2c37","Type":"ContainerStarted","Data":"824a40d9d4acd0de8558b3ed369788999758711f00265b6c5c9b028e0c2e5c95"}
Oct 30 13:48:58.725612 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:58.725551    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz" podStartSLOduration=2.725534846 podStartE2EDuration="2.725534846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:58.7238315 +0000 UTC m=+4603.199551112" watchObservedRunningTime="2026-10-30 13:48:58.725534846 +0000 UTC m=+4603.201254456"
Oct 30 13:48:59.033014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:59.032975    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"]
Oct 30 13:48:59.037014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:59.036974    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99dd107-j7lqt"]
Oct 30 13:48:59.287148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:59.287059    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:48:59.694462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:48:59.694381    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833a70fb-536c-4120-b052-9602e8ebe0ac" path="/var/lib/kubelet/pods/833a70fb-536c-4120-b052-9602e8ebe0ac/volumes"
Oct 30 13:49:01.106654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:01.106611    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:49:01.120125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:01.120083    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:49:01.729048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:01.729014    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:49:06.068247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:06.068187    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"]
Oct 30 13:49:06.086103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:06.086059    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ece70c9-zbmx5"]
Oct 30 13:49:07.286702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:07.286664    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:49:07.299672 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:07.299646    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:49:07.703574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:07.703480    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44c27aa-73bc-490f-9865-eb4fb8f42bbd" path="/var/lib/kubelet/pods/f44c27aa-73bc-490f-9865-eb4fb8f42bbd/volumes"
Oct 30 13:49:07.790723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:07.790687    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:49:08.570456 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:49:08.570399    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:49:21.372344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.372291    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"]
Oct 30 13:49:21.378467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.378439    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.417486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.417432    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-92855c\"/\"default-dockercfg-42w2p\""
Oct 30 13:49:21.417830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.417486    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-92855c\"/\"kube-root-ca.crt\""
Oct 30 13:49:21.417830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.417517    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-92855c\"/\"openshift-service-ca.crt\""
Oct 30 13:49:21.417830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.417587    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-92855c\"/\"sub-test-model-92855c-kserve-self-signed-certs\""
Oct 30 13:49:21.423440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.423405    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"]
Oct 30 13:49:21.440466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.440431    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-model-cache\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.440636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.440528    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-home\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.440636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.440579    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-dshm\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.440785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.440761    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-tmp-dir\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.440849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.440818    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssl26\" (UniqueName: \"kubernetes.io/projected/82dab4c8-de9b-46dd-812d-40f6c8ded436-kube-api-access-ssl26\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.440902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.440887    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82dab4c8-de9b-46dd-812d-40f6c8ded436-tls-certs\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.541937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.541899    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-tmp-dir\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.542146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.542020    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssl26\" (UniqueName: \"kubernetes.io/projected/82dab4c8-de9b-46dd-812d-40f6c8ded436-kube-api-access-ssl26\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.542146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.542124    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82dab4c8-de9b-46dd-812d-40f6c8ded436-tls-certs\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.542348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.542265    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-model-cache\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.542348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.542330    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-home\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.542497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.542372    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-dshm\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.542497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.542404    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-tmp-dir\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.542637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.542615    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-model-cache\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.542704 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.542668    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-home\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.544993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.544961    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-dshm\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.545175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.545153    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82dab4c8-de9b-46dd-812d-40f6c8ded436-tls-certs\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.566179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.566139    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssl26\" (UniqueName: \"kubernetes.io/projected/82dab4c8-de9b-46dd-812d-40f6c8ded436-kube-api-access-ssl26\") pod \"sub-test-model-92855c-kserve-57844fd985-65xhp\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") " pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:21.723285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:21.719528    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:22.146377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:22.146333    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"]
Oct 30 13:49:22.147865 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:49:22.147835    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82dab4c8_de9b_46dd_812d_40f6c8ded436.slice/crio-c314ea748674fde7b4134aa1597f51139de978b3dd7c0b6e8441f46967a55f3f WatchSource:0}: Error finding container c314ea748674fde7b4134aa1597f51139de978b3dd7c0b6e8441f46967a55f3f: Status 404 returned error can't find the container with id c314ea748674fde7b4134aa1597f51139de978b3dd7c0b6e8441f46967a55f3f
Oct 30 13:49:22.943767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:22.943725    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp" event={"ID":"82dab4c8-de9b-46dd-812d-40f6c8ded436","Type":"ContainerStarted","Data":"d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a"}
Oct 30 13:49:22.943767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:22.943774    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp" event={"ID":"82dab4c8-de9b-46dd-812d-40f6c8ded436","Type":"ContainerStarted","Data":"c314ea748674fde7b4134aa1597f51139de978b3dd7c0b6e8441f46967a55f3f"}
Oct 30 13:49:22.989752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:22.989682    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp" podStartSLOduration=1.989668311 podStartE2EDuration="1.989668311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:22.986390161 +0000 UTC m=+4627.462109783" watchObservedRunningTime="2026-10-30 13:49:22.989668311 +0000 UTC m=+4627.465387921"
Oct 30 13:49:23.719695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:23.719651    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:27.031465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:27.031421    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"]
Oct 30 13:49:27.043092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:27.043051    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1f3bc1-d8z67"]
Oct 30 13:49:27.710730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:27.710675    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5ec855-5930-44b7-9218-3449d29dd739" path="/var/lib/kubelet/pods/6f5ec855-5930-44b7-9218-3449d29dd739/volumes"
Oct 30 13:49:31.782715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:31.782675    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:32.722413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:32.722353    4993 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp" podUID="82dab4c8-de9b-46dd-812d-40f6c8ded436" containerName="main" probeResult="failure" output="Get \"https://10.132.1.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:49:32.742236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:32.740298    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:49:32.742236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:32.741627    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:49:32.742911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:32.742871    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:49:33.086359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:33.086306    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:49:33.087391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:33.087355    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 30 13:49:33.186674 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:33.186620    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:49:38.573065 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:49:38.573021    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:49:39.701091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.701049    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"]
Oct 30 13:49:39.705712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.705687    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.720712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.720686    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:49:39.732583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.731977    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:49:39.732583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.732033    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:49:39.732583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.732046    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:49:39.742176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.742143    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"]
Oct 30 13:49:39.829641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.829591    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86jk2\" (UniqueName: \"kubernetes.io/projected/7240202d-ce7f-4105-81d2-07a6ca5e109b-kube-api-access-86jk2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.829834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.829721    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.829834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.829803    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.931409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.931368    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86jk2\" (UniqueName: \"kubernetes.io/projected/7240202d-ce7f-4105-81d2-07a6ca5e109b-kube-api-access-86jk2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.931625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.931437    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.931625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.931492    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.932241 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.932188    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.933970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.933946    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:39.955240 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:39.955127    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86jk2\" (UniqueName: \"kubernetes.io/projected/7240202d-ce7f-4105-81d2-07a6ca5e109b-kube-api-access-86jk2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:40.039418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:40.039346    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:40.394971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:40.394936    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"]
Oct 30 13:49:40.396412 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:49:40.396374    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7240202d_ce7f_4105_81d2_07a6ca5e109b.slice/crio-b8db6b67a30d844273146d488d0d6377feeea6b5544da92be79a2783b497775e WatchSource:0}: Error finding container b8db6b67a30d844273146d488d0d6377feeea6b5544da92be79a2783b497775e: Status 404 returned error can't find the container with id b8db6b67a30d844273146d488d0d6377feeea6b5544da92be79a2783b497775e
Oct 30 13:49:41.253973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:41.253929    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9" event={"ID":"7240202d-ce7f-4105-81d2-07a6ca5e109b","Type":"ContainerStarted","Data":"cb4eb7051b9694cc180f952b172beb58f0ee23b780267cc2c1be14835e409dc1"}
Oct 30 13:49:41.254536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:41.254023    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9" event={"ID":"7240202d-ce7f-4105-81d2-07a6ca5e109b","Type":"ContainerStarted","Data":"b8db6b67a30d844273146d488d0d6377feeea6b5544da92be79a2783b497775e"}
Oct 30 13:49:41.309140 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:41.309071    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9" podStartSLOduration=2.309054263 podStartE2EDuration="2.309054263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:41.307818412 +0000 UTC m=+4645.783538024" watchObservedRunningTime="2026-10-30 13:49:41.309054263 +0000 UTC m=+4645.784773874"
Oct 30 13:49:41.732799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:41.732751    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:41.779851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:41.779524    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:49:42.264555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:42.264513    4993 generic.go:356] "Generic (PLEG): container finished" podID="7240202d-ce7f-4105-81d2-07a6ca5e109b" containerID="cb4eb7051b9694cc180f952b172beb58f0ee23b780267cc2c1be14835e409dc1" exitCode=0
Oct 30 13:49:42.265082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:42.264598    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9" event={"ID":"7240202d-ce7f-4105-81d2-07a6ca5e109b","Type":"ContainerDied","Data":"cb4eb7051b9694cc180f952b172beb58f0ee23b780267cc2c1be14835e409dc1"}
Oct 30 13:49:43.031071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.031032    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"]
Oct 30 13:49:43.042819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.042779    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c777df28-9sxgd"]
Oct 30 13:49:43.497279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.497248    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:43.529098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.529067    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-openshift-ca\") pod \"7240202d-ce7f-4105-81d2-07a6ca5e109b\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") "
Oct 30 13:49:43.529098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.529114    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7240202d-ce7f-4105-81d2-07a6ca5e109b-kube-api-access-86jk2\" (UniqueName: \"kubernetes.io/projected/7240202d-ce7f-4105-81d2-07a6ca5e109b-kube-api-access-86jk2\") pod \"7240202d-ce7f-4105-81d2-07a6ca5e109b\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") "
Oct 30 13:49:43.529391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.529238    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-xks-ca\") pod \"7240202d-ce7f-4105-81d2-07a6ca5e109b\" (UID: \"7240202d-ce7f-4105-81d2-07a6ca5e109b\") "
Oct 30 13:49:43.529487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.529456    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-openshift-ca" pod "7240202d-ce7f-4105-81d2-07a6ca5e109b" (UID: "7240202d-ce7f-4105-81d2-07a6ca5e109b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:43.529938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.529913    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:49:43.531343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.531313    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-xks-ca" pod "7240202d-ce7f-4105-81d2-07a6ca5e109b" (UID: "7240202d-ce7f-4105-81d2-07a6ca5e109b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:49:43.531471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.531454    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7240202d-ce7f-4105-81d2-07a6ca5e109b-kube-api-access-86jk2" pod "7240202d-ce7f-4105-81d2-07a6ca5e109b" (UID: "7240202d-ce7f-4105-81d2-07a6ca5e109b"). InnerVolumeSpecName "kube-api-access-86jk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:43.631003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.630910    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86jk2\" (UniqueName: \"kubernetes.io/projected/7240202d-ce7f-4105-81d2-07a6ca5e109b-kube-api-access-86jk2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:49:43.631003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.630947    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240202d-ce7f-4105-81d2-07a6ca5e109b-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:49:43.692775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:43.692733    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc565eb2-e27c-433a-a5a4-0e6219ea7670" path="/var/lib/kubelet/pods/cc565eb2-e27c-433a-a5a4-0e6219ea7670/volumes"
Oct 30 13:49:44.287440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:44.287396    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"
Oct 30 13:49:44.287440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:44.287412    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9" event={"ID":"7240202d-ce7f-4105-81d2-07a6ca5e109b","Type":"ContainerDied","Data":"b8db6b67a30d844273146d488d0d6377feeea6b5544da92be79a2783b497775e"}
Oct 30 13:49:44.287717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:44.287505    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8db6b67a30d844273146d488d0d6377feeea6b5544da92be79a2783b497775e"
Oct 30 13:49:55.039145 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:55.039097    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"]
Oct 30 13:49:55.049524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:55.049478    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc1258a-sv2w6"]
Oct 30 13:49:55.693589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:55.693553    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced2627f-1550-49aa-b455-ac3581ef9ef6" path="/var/lib/kubelet/pods/ced2627f-1550-49aa-b455-ac3581ef9ef6/volumes"
Oct 30 13:49:57.033359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:57.033306    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"]
Oct 30 13:49:57.041690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:57.041656    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4cce218-rbn7d"]
Oct 30 13:49:57.693805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:49:57.693764    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3537761-041e-4371-9f16-546ceceae4d6" path="/var/lib/kubelet/pods/b3537761-041e-4371-9f16-546ceceae4d6/volumes"
Oct 30 13:50:01.633359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:01.633317    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"]
Oct 30 13:50:01.644591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:01.644553    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0505b0-tfwk9"]
Oct 30 13:50:01.697023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:01.696985    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7240202d-ce7f-4105-81d2-07a6ca5e109b" path="/var/lib/kubelet/pods/7240202d-ce7f-4105-81d2-07a6ca5e109b/volumes"
Oct 30 13:50:08.575768 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:50:08.575709    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:50:09.052477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.052434    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"]
Oct 30 13:50:09.053107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.053089    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7240202d-ce7f-4105-81d2-07a6ca5e109b" containerName="revoke-keys"
Oct 30 13:50:09.053107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.053107    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7240202d-ce7f-4105-81d2-07a6ca5e109b" containerName="revoke-keys"
Oct 30 13:50:09.053261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.053226    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7240202d-ce7f-4105-81d2-07a6ca5e109b" containerName="revoke-keys"
Oct 30 13:50:09.057661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.057638    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.087337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.087282    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:50:09.087667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.087535    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:50:09.087829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.087602    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:50:09.097849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.097805    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"]
Oct 30 13:50:09.099375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.099344    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:50:09.124832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.124793    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.125054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.124839    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.125054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.124975    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl82g\" (UniqueName: \"kubernetes.io/projected/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-kube-api-access-bl82g\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.226677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.226633    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl82g\" (UniqueName: \"kubernetes.io/projected/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-kube-api-access-bl82g\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.226884 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.226862    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.226931 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.226894    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.227730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.227703    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.229439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.229417    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.250903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.250869    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl82g\" (UniqueName: \"kubernetes.io/projected/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-kube-api-access-bl82g\") pod \"maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.374154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.374041    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:09.689302 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:50:09.689267    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9af1cb_b3c4_492c_92de_9f1f42605f7f.slice/crio-5799391d487770ae1a3b7ce90365dbabcfe5aa8b2429f9cd84b9b828e8e0f7c2 WatchSource:0}: Error finding container 5799391d487770ae1a3b7ce90365dbabcfe5aa8b2429f9cd84b9b828e8e0f7c2: Status 404 returned error can't find the container with id 5799391d487770ae1a3b7ce90365dbabcfe5aa8b2429f9cd84b9b828e8e0f7c2
Oct 30 13:50:09.697482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:09.697448    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"]
Oct 30 13:50:10.637230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:10.636360    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z" event={"ID":"4e9af1cb-b3c4-492c-92de-9f1f42605f7f","Type":"ContainerStarted","Data":"3e74de7c6f00202accd657de81822c6c7f651ab53e778f9afbc39d72111e8e9f"}
Oct 30 13:50:10.637230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:10.636461    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z" event={"ID":"4e9af1cb-b3c4-492c-92de-9f1f42605f7f","Type":"ContainerStarted","Data":"5799391d487770ae1a3b7ce90365dbabcfe5aa8b2429f9cd84b9b828e8e0f7c2"}
Oct 30 13:50:10.682534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:10.682162    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z" podStartSLOduration=1.682144934 podStartE2EDuration="1.682144934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:10.678367104 +0000 UTC m=+4675.154086715" watchObservedRunningTime="2026-10-30 13:50:10.682144934 +0000 UTC m=+4675.157864545"
Oct 30 13:50:10.968134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:10.968030    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"]
Oct 30 13:50:10.972244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:10.972198    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:10.999469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:10.999417    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"]
Oct 30 13:50:11.095038 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.094981    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.095339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.095081    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.107228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.104298    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pl97\" (UniqueName: \"kubernetes.io/projected/94e7469d-8d5c-447d-8254-536398bb6d2c-kube-api-access-2pl97\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.205554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.205505    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.205781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.205577    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.205781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.205697    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pl97\" (UniqueName: \"kubernetes.io/projected/94e7469d-8d5c-447d-8254-536398bb6d2c-kube-api-access-2pl97\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.206313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.206281    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.208116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.208095    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.231959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.231922    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pl97\" (UniqueName: \"kubernetes.io/projected/94e7469d-8d5c-447d-8254-536398bb6d2c-kube-api-access-2pl97\") pod \"maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.287957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.287912    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:11.612134 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:50:11.612102    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e7469d_8d5c_447d_8254_536398bb6d2c.slice/crio-54d461735f603ed9f828f8c96678620fdd92badf7904b9bfc14a304ed7db8493 WatchSource:0}: Error finding container 54d461735f603ed9f828f8c96678620fdd92badf7904b9bfc14a304ed7db8493: Status 404 returned error can't find the container with id 54d461735f603ed9f828f8c96678620fdd92badf7904b9bfc14a304ed7db8493
Oct 30 13:50:11.613194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.613164    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"]
Oct 30 13:50:11.648129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.648095    4993 generic.go:356] "Generic (PLEG): container finished" podID="4e9af1cb-b3c4-492c-92de-9f1f42605f7f" containerID="3e74de7c6f00202accd657de81822c6c7f651ab53e778f9afbc39d72111e8e9f" exitCode=0
Oct 30 13:50:11.648305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.648168    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z" event={"ID":"4e9af1cb-b3c4-492c-92de-9f1f42605f7f","Type":"ContainerDied","Data":"3e74de7c6f00202accd657de81822c6c7f651ab53e778f9afbc39d72111e8e9f"}
Oct 30 13:50:11.649803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:11.649777    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5" event={"ID":"94e7469d-8d5c-447d-8254-536398bb6d2c","Type":"ContainerStarted","Data":"54d461735f603ed9f828f8c96678620fdd92badf7904b9bfc14a304ed7db8493"}
Oct 30 13:50:12.660756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.660703    4993 generic.go:356] "Generic (PLEG): container finished" podID="94e7469d-8d5c-447d-8254-536398bb6d2c" containerID="1af4e1d3514f390aa749f0c72cf64786ae6e9f40465523be34bd7d53fb8b1193" exitCode=0
Oct 30 13:50:12.661309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.660784    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5" event={"ID":"94e7469d-8d5c-447d-8254-536398bb6d2c","Type":"ContainerDied","Data":"1af4e1d3514f390aa749f0c72cf64786ae6e9f40465523be34bd7d53fb8b1193"}
Oct 30 13:50:12.843601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.843566    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:12.929201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.929158    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-openshift-ca\") pod \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") "
Oct 30 13:50:12.929430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.929281    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-kube-api-access-bl82g\" (UniqueName: \"kubernetes.io/projected/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-kube-api-access-bl82g\") pod \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") "
Oct 30 13:50:12.929430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.929376    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-xks-ca\") pod \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\" (UID: \"4e9af1cb-b3c4-492c-92de-9f1f42605f7f\") "
Oct 30 13:50:12.929597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.929571    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-openshift-ca" pod "4e9af1cb-b3c4-492c-92de-9f1f42605f7f" (UID: "4e9af1cb-b3c4-492c-92de-9f1f42605f7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:12.930029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.930006    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:50:12.931664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.931641    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-xks-ca" pod "4e9af1cb-b3c4-492c-92de-9f1f42605f7f" (UID: "4e9af1cb-b3c4-492c-92de-9f1f42605f7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:12.931732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:12.931702    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-kube-api-access-bl82g" pod "4e9af1cb-b3c4-492c-92de-9f1f42605f7f" (UID: "4e9af1cb-b3c4-492c-92de-9f1f42605f7f"). InnerVolumeSpecName "kube-api-access-bl82g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:13.031582 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:13.031532    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl82g\" (UniqueName: \"kubernetes.io/projected/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-kube-api-access-bl82g\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:50:13.031582 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:13.031574    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9af1cb-b3c4-492c-92de-9f1f42605f7f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:50:13.674228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:13.674123    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"
Oct 30 13:50:13.674228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:13.674152    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z" event={"ID":"4e9af1cb-b3c4-492c-92de-9f1f42605f7f","Type":"ContainerDied","Data":"5799391d487770ae1a3b7ce90365dbabcfe5aa8b2429f9cd84b9b828e8e0f7c2"}
Oct 30 13:50:13.674737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:13.674272    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5799391d487770ae1a3b7ce90365dbabcfe5aa8b2429f9cd84b9b828e8e0f7c2"
Oct 30 13:50:13.869260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:13.869187    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:14.050075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.050030    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e7469d-8d5c-447d-8254-536398bb6d2c-kube-api-access-2pl97\" (UniqueName: \"kubernetes.io/projected/94e7469d-8d5c-447d-8254-536398bb6d2c-kube-api-access-2pl97\") pod \"94e7469d-8d5c-447d-8254-536398bb6d2c\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") "
Oct 30 13:50:14.050287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.050082    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-openshift-ca\") pod \"94e7469d-8d5c-447d-8254-536398bb6d2c\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") "
Oct 30 13:50:14.050359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.050342    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-xks-ca\") pod \"94e7469d-8d5c-447d-8254-536398bb6d2c\" (UID: \"94e7469d-8d5c-447d-8254-536398bb6d2c\") "
Oct 30 13:50:14.050546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.050517    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-openshift-ca" pod "94e7469d-8d5c-447d-8254-536398bb6d2c" (UID: "94e7469d-8d5c-447d-8254-536398bb6d2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:14.050973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.050887    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:50:14.052549 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.052520    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e7469d-8d5c-447d-8254-536398bb6d2c-kube-api-access-2pl97" pod "94e7469d-8d5c-447d-8254-536398bb6d2c" (UID: "94e7469d-8d5c-447d-8254-536398bb6d2c"). InnerVolumeSpecName "kube-api-access-2pl97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:14.052734 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.052708    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-xks-ca" pod "94e7469d-8d5c-447d-8254-536398bb6d2c" (UID: "94e7469d-8d5c-447d-8254-536398bb6d2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:14.151996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.151956    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e7469d-8d5c-447d-8254-536398bb6d2c-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:50:14.151996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.151995    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pl97\" (UniqueName: \"kubernetes.io/projected/94e7469d-8d5c-447d-8254-536398bb6d2c-kube-api-access-2pl97\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:50:14.686282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.686248    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"
Oct 30 13:50:14.686826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.686280    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5" event={"ID":"94e7469d-8d5c-447d-8254-536398bb6d2c","Type":"ContainerDied","Data":"54d461735f603ed9f828f8c96678620fdd92badf7904b9bfc14a304ed7db8493"}
Oct 30 13:50:14.686826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:14.686341    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d461735f603ed9f828f8c96678620fdd92badf7904b9bfc14a304ed7db8493"
Oct 30 13:50:19.543339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:19.543259    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"]
Oct 30 13:50:23.342661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:23.342622    4993 scope.go:122] "RemoveContainer" containerID="9bd390d8d4c61416f0b384faf3a9c9a58b157732b2632cfd8b600300336d5a6a"
Oct 30 13:50:23.357070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:23.357036    4993 scope.go:122] "RemoveContainer" containerID="820d0b7fd974d0078c541cbef8314822535c51c23ad0a32890c5293235fbc1bb"
Oct 30 13:50:23.370317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:23.370293    4993 scope.go:122] "RemoveContainer" containerID="4d94a75fedfa7f2d38bdf1a54c2b923598a14eaeaee7c9e3b6ccdbf55b4fa646"
Oct 30 13:50:23.381737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:23.381711    4993 scope.go:122] "RemoveContainer" containerID="3805f25351a8ed2d0b80a8db62772e8af8b0abbf7ac3bd3df2822af1840aea5d"
Oct 30 13:50:23.395031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:23.394999    4993 scope.go:122] "RemoveContainer" containerID="71e455212a767156494b0bbce28b2e9f5db0e3ac2a751e477b1a53529941ab67"
Oct 30 13:50:24.544473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.544391    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp" podUID="2af20f1b-1631-484d-8f4d-6dc472486867" containerName="payload-pre-processing" containerID="cri-o://403d20828c15358435fefb84b1d1596b098a8fda4292e55c94ab1b5f278c74b5" gracePeriod=25
Oct 30 13:50:24.812059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.811958    4993 generic.go:356] "Generic (PLEG): container finished" podID="2af20f1b-1631-484d-8f4d-6dc472486867" containerID="403d20828c15358435fefb84b1d1596b098a8fda4292e55c94ab1b5f278c74b5" exitCode=0
Oct 30 13:50:24.812257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.812108    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp" event={"ID":"2af20f1b-1631-484d-8f4d-6dc472486867","Type":"ContainerDied","Data":"403d20828c15358435fefb84b1d1596b098a8fda4292e55c94ab1b5f278c74b5"}
Oct 30 13:50:24.893523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.893480    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:50:24.944790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.944746    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af20f1b-1631-484d-8f4d-6dc472486867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af20f1b-1631-484d-8f4d-6dc472486867-plugins-config-volume\") pod \"2af20f1b-1631-484d-8f4d-6dc472486867\" (UID: \"2af20f1b-1631-484d-8f4d-6dc472486867\") "
Oct 30 13:50:24.944991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.944812    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af20f1b-1631-484d-8f4d-6dc472486867-kube-api-access-lcvtg\" (UniqueName: \"kubernetes.io/projected/2af20f1b-1631-484d-8f4d-6dc472486867-kube-api-access-lcvtg\") pod \"2af20f1b-1631-484d-8f4d-6dc472486867\" (UID: \"2af20f1b-1631-484d-8f4d-6dc472486867\") "
Oct 30 13:50:24.945156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.945130    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af20f1b-1631-484d-8f4d-6dc472486867-plugins-config-volume" pod "2af20f1b-1631-484d-8f4d-6dc472486867" (UID: "2af20f1b-1631-484d-8f4d-6dc472486867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:24.945693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.945660    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af20f1b-1631-484d-8f4d-6dc472486867-plugins-config-volume\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:50:24.947369 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:24.947319    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af20f1b-1631-484d-8f4d-6dc472486867-kube-api-access-lcvtg" pod "2af20f1b-1631-484d-8f4d-6dc472486867" (UID: "2af20f1b-1631-484d-8f4d-6dc472486867"). InnerVolumeSpecName "kube-api-access-lcvtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:25.047139 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:25.047090    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcvtg\" (UniqueName: \"kubernetes.io/projected/2af20f1b-1631-484d-8f4d-6dc472486867-kube-api-access-lcvtg\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:50:25.821838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:25.821800    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"
Oct 30 13:50:25.822300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:25.821810    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp" event={"ID":"2af20f1b-1631-484d-8f4d-6dc472486867","Type":"ContainerDied","Data":"6185c0f3bfa9a02acaa6d527fb24bbb408fb263ac971f16c1ecd83cd1c81adee"}
Oct 30 13:50:25.822300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:25.821939    4993 scope.go:122] "RemoveContainer" containerID="403d20828c15358435fefb84b1d1596b098a8fda4292e55c94ab1b5f278c74b5"
Oct 30 13:50:25.874766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:25.874724    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"]
Oct 30 13:50:25.888665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:25.888620    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d01cd5a5-8fcbbf4d7-wp4tp"]
Oct 30 13:50:27.694554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:27.694510    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af20f1b-1631-484d-8f4d-6dc472486867" path="/var/lib/kubelet/pods/2af20f1b-1631-484d-8f4d-6dc472486867/volumes"
Oct 30 13:50:38.569413 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:50:38.569358    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:50:51.181466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:51.181424    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"]
Oct 30 13:50:51.193063 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:51.193030    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d01cd5a5-v4t9z"]
Oct 30 13:50:51.694914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:51.694863    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9af1cb-b3c4-492c-92de-9f1f42605f7f" path="/var/lib/kubelet/pods/4e9af1cb-b3c4-492c-92de-9f1f42605f7f/volumes"
Oct 30 13:50:57.143696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:57.143656    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"]
Oct 30 13:50:57.152492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:57.152454    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0d9115-9ncf5"]
Oct 30 13:50:57.694249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:50:57.694197    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e7469d-8d5c-447d-8254-536398bb6d2c" path="/var/lib/kubelet/pods/94e7469d-8d5c-447d-8254-536398bb6d2c/volumes"
Oct 30 13:51:00.832539 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:00.832486    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"]
Oct 30 13:51:00.833708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:00.833626    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy" containerID="cri-o://88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301" gracePeriod=30
Oct 30 13:51:05.020808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.020767    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"]
Oct 30 13:51:05.021366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.021028    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy" containerID="cri-o://8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf" gracePeriod=30
Oct 30 13:51:05.744850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.744808    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"]
Oct 30 13:51:05.745635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745617    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4e9af1cb-b3c4-492c-92de-9f1f42605f7f" containerName="revoke-keys"
Oct 30 13:51:05.745698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745638    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9af1cb-b3c4-492c-92de-9f1f42605f7f" containerName="revoke-keys"
Oct 30 13:51:05.745698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745671    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2af20f1b-1631-484d-8f4d-6dc472486867" containerName="payload-pre-processing"
Oct 30 13:51:05.745698 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745680    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af20f1b-1631-484d-8f4d-6dc472486867" containerName="payload-pre-processing"
Oct 30 13:51:05.745810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745702    4993 cpu_manager.go:414] "containerMap: removing container" podUID="94e7469d-8d5c-447d-8254-536398bb6d2c" containerName="revoke-keys"
Oct 30 13:51:05.745810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745711    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e7469d-8d5c-447d-8254-536398bb6d2c" containerName="revoke-keys"
Oct 30 13:51:05.745883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745865    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af20f1b-1631-484d-8f4d-6dc472486867" containerName="payload-pre-processing"
Oct 30 13:51:05.745883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745879    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e7469d-8d5c-447d-8254-536398bb6d2c" containerName="revoke-keys"
Oct 30 13:51:05.745948 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.745898    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9af1cb-b3c4-492c-92de-9f1f42605f7f" containerName="revoke-keys"
Oct 30 13:51:05.749906 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.749883    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.778323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.778285    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:51:05.778323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.778296    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:51:05.783447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.783405    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"]
Oct 30 13:51:05.785370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.785332    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:51:05.785578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.785554    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:51:05.835555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.835511    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.835804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.835785    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.835871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.835818    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjwx5\" (UniqueName: \"kubernetes.io/projected/2a86cdcd-ede9-4475-8c24-5518364f7b64-kube-api-access-kjwx5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.937359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.937322    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.937562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.937371    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjwx5\" (UniqueName: \"kubernetes.io/projected/2a86cdcd-ede9-4475-8c24-5518364f7b64-kube-api-access-kjwx5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.937562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.937446    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.938187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.938158    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.939915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.939893    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:05.960840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:05.960804    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjwx5\" (UniqueName: \"kubernetes.io/projected/2a86cdcd-ede9-4475-8c24-5518364f7b64-kube-api-access-kjwx5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:06.065329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.065289    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:06.126459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.126423    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.262469    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-token\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-token\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.262656    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-envoy\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.262713    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-kube-api-access-bgnhw\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.262769    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-credential-socket\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.262810    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istiod-ca-cert\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.262848    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-data\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.262950    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-socket\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.263002    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-podinfo\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.263045    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-certs\") pod \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\" (UID: \"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6\") "
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.264256    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-credential-socket" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.264795    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-data" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.265194    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istiod-ca-cert" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.265467    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-socket" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:06.266259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.265859    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-certs" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:06.274814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.273965    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-kube-api-access-bgnhw" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "kube-api-access-bgnhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:06.279966 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.279733    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-podinfo" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:51:06.279966 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.279900    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-token" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:06.280507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.280379    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-envoy" pod "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" (UID: "6264a5b2-8654-48bc-a5e8-9d26c7e6acd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:06.364892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364852    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.364892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364893    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-kube-api-access-bgnhw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.365134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364904    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.365134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364913    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.365134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364923    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.365134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364932    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.365134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364943    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.365134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364957    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.365134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.364968    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.365134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.365113    4993 generic.go:356] "Generic (PLEG): container finished" podID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerID="88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301" exitCode=0
Oct 30 13:51:06.365538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.365179    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"
Oct 30 13:51:06.365538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.365201    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" event={"ID":"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6","Type":"ContainerDied","Data":"88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301"}
Oct 30 13:51:06.365538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.365279    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh" event={"ID":"6264a5b2-8654-48bc-a5e8-9d26c7e6acd6","Type":"ContainerDied","Data":"0af9bee3864633400c3c2591560f426c66446ca3e36704d4ab7d6570c361b48f"}
Oct 30 13:51:06.365538 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.365305    4993 scope.go:122] "RemoveContainer" containerID="88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301"
Oct 30 13:51:06.379849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.379815    4993 scope.go:122] "RemoveContainer" containerID="61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052"
Oct 30 13:51:06.391422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.391396    4993 scope.go:122] "RemoveContainer" containerID="88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301"
Oct 30 13:51:06.391820 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:51:06.391798    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301\": container with ID starting with 88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301 not found: ID does not exist" containerID="88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301"
Oct 30 13:51:06.391871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.391832    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301"} err="failed to get container status \"88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301\": rpc error: code = NotFound desc = could not find container \"88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301\": container with ID starting with 88643c62e94a0768b68d440fe072454b71aec1ee8306a778832084312ccfc301 not found: ID does not exist"
Oct 30 13:51:06.391871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.391852    4993 scope.go:122] "RemoveContainer" containerID="61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052"
Oct 30 13:51:06.392154 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:51:06.392125    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052\": container with ID starting with 61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052 not found: ID does not exist" containerID="61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052"
Oct 30 13:51:06.392254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.392164    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052"} err="failed to get container status \"61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052\": rpc error: code = NotFound desc = could not find container \"61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052\": container with ID starting with 61ef8744a39edb8648275965fdf86fefe385eb266ea052e13a8a7a24247db052 not found: ID does not exist"
Oct 30 13:51:06.418134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.418095    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"]
Oct 30 13:51:06.426339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.426294    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d01cd5a5-openshift-default-df545ffd9-f8mrh"]
Oct 30 13:51:06.453456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:06.453426    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"]
Oct 30 13:51:06.457237 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:51:06.457185    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a86cdcd_ede9_4475_8c24_5518364f7b64.slice/crio-ca548f1812e148492d604fc1cc2748c35bd528e3a74053b079b83f055d962dd4 WatchSource:0}: Error finding container ca548f1812e148492d604fc1cc2748c35bd528e3a74053b079b83f055d962dd4: Status 404 returned error can't find the container with id ca548f1812e148492d604fc1cc2748c35bd528e3a74053b079b83f055d962dd4
Oct 30 13:51:07.375510 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:07.375468    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd" event={"ID":"2a86cdcd-ede9-4475-8c24-5518364f7b64","Type":"ContainerStarted","Data":"95e20b4e57bb0be03468c0f2b261ddc3e17b3f61fa24d4e6aca95f0e3b49147b"}
Oct 30 13:51:07.376115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:07.376085    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd" event={"ID":"2a86cdcd-ede9-4475-8c24-5518364f7b64","Type":"ContainerStarted","Data":"ca548f1812e148492d604fc1cc2748c35bd528e3a74053b079b83f055d962dd4"}
Oct 30 13:51:07.423267 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:07.423190    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd" podStartSLOduration=2.423174993 podStartE2EDuration="2.423174993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:07.420907362 +0000 UTC m=+4731.896626972" watchObservedRunningTime="2026-10-30 13:51:07.423174993 +0000 UTC m=+4731.898894644"
Oct 30 13:51:07.693832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:07.693752    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" path="/var/lib/kubelet/pods/6264a5b2-8654-48bc-a5e8-9d26c7e6acd6/volumes"
Oct 30 13:51:08.397488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:08.397440    4993 generic.go:356] "Generic (PLEG): container finished" podID="2a86cdcd-ede9-4475-8c24-5518364f7b64" containerID="95e20b4e57bb0be03468c0f2b261ddc3e17b3f61fa24d4e6aca95f0e3b49147b" exitCode=0
Oct 30 13:51:08.398003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:08.397737    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd" event={"ID":"2a86cdcd-ede9-4475-8c24-5518364f7b64","Type":"ContainerDied","Data":"95e20b4e57bb0be03468c0f2b261ddc3e17b3f61fa24d4e6aca95f0e3b49147b"}
Oct 30 13:51:08.571777 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:51:08.571727    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:51:09.637721 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.637687    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:09.711554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.711425    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a86cdcd-ede9-4475-8c24-5518364f7b64-kube-api-access-kjwx5\" (UniqueName: \"kubernetes.io/projected/2a86cdcd-ede9-4475-8c24-5518364f7b64-kube-api-access-kjwx5\") pod \"2a86cdcd-ede9-4475-8c24-5518364f7b64\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") "
Oct 30 13:51:09.711747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.711633    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-openshift-ca\") pod \"2a86cdcd-ede9-4475-8c24-5518364f7b64\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") "
Oct 30 13:51:09.712094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.712054    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-openshift-ca" pod "2a86cdcd-ede9-4475-8c24-5518364f7b64" (UID: "2a86cdcd-ede9-4475-8c24-5518364f7b64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:09.712260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.712116    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-xks-ca\") pod \"2a86cdcd-ede9-4475-8c24-5518364f7b64\" (UID: \"2a86cdcd-ede9-4475-8c24-5518364f7b64\") "
Oct 30 13:51:09.712824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.712797    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:09.715395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.715362    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a86cdcd-ede9-4475-8c24-5518364f7b64-kube-api-access-kjwx5" pod "2a86cdcd-ede9-4475-8c24-5518364f7b64" (UID: "2a86cdcd-ede9-4475-8c24-5518364f7b64"). InnerVolumeSpecName "kube-api-access-kjwx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:09.716982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.716933    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-xks-ca" pod "2a86cdcd-ede9-4475-8c24-5518364f7b64" (UID: "2a86cdcd-ede9-4475-8c24-5518364f7b64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:51:09.814302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.814252    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a86cdcd-ede9-4475-8c24-5518364f7b64-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:09.814302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:09.814298    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjwx5\" (UniqueName: \"kubernetes.io/projected/2a86cdcd-ede9-4475-8c24-5518364f7b64-kube-api-access-kjwx5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.295708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.295622    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:51:10.419497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.419443    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"
Oct 30 13:51:10.419497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.419484    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd" event={"ID":"2a86cdcd-ede9-4475-8c24-5518364f7b64","Type":"ContainerDied","Data":"ca548f1812e148492d604fc1cc2748c35bd528e3a74053b079b83f055d962dd4"}
Oct 30 13:51:10.419796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.419548    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca548f1812e148492d604fc1cc2748c35bd528e3a74053b079b83f055d962dd4"
Oct 30 13:51:10.422367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422337    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-envoy\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.422547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422372    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istiod-ca-cert\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.422547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422447    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-token\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-token\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.422547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422507    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-kube-api-access-h28kz\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-kube-api-access-h28kz\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.422738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422577    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-data\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.422738 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422665    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-podinfo\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.422852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422757    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-credential-socket\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.422852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422835    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-socket\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.422958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422874    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istiod-ca-cert" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:10.422958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.422886    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-certs\") pod \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\" (UID: \"7b5ff6f0-dce9-4419-aa9e-abe3045212f7\") "
Oct 30 13:51:10.423152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423128    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-certs" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:10.423395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423371    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-socket" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:10.423555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423529    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-credential-socket" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:10.423628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423567    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-data" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:10.423756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423726    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.423756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423752    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.423893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423766    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.423893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423781    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.423893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.423796    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.424757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.424727    4993 generic.go:356] "Generic (PLEG): container finished" podID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerID="8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf" exitCode=0
Oct 30 13:51:10.424897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.424802    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"
Oct 30 13:51:10.424897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.424826    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" event={"ID":"7b5ff6f0-dce9-4419-aa9e-abe3045212f7","Type":"ContainerDied","Data":"8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf"}
Oct 30 13:51:10.424897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.424881    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c" event={"ID":"7b5ff6f0-dce9-4419-aa9e-abe3045212f7","Type":"ContainerDied","Data":"50a6712885b75d7c74dee0ebc03da520f47b5bafce5fb33fbf81d9b82f56eda5"}
Oct 30 13:51:10.425094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.424914    4993 scope.go:122] "RemoveContainer" containerID="8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf"
Oct 30 13:51:10.425407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.425379    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-envoy" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:10.425504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.425426    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-token" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:10.425617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.425589    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-kube-api-access-h28kz" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "kube-api-access-h28kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:10.426278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.426192    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-podinfo" pod "7b5ff6f0-dce9-4419-aa9e-abe3045212f7" (UID: "7b5ff6f0-dce9-4419-aa9e-abe3045212f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:51:10.438148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.438121    4993 scope.go:122] "RemoveContainer" containerID="1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd"
Oct 30 13:51:10.449922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.449897    4993 scope.go:122] "RemoveContainer" containerID="8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf"
Oct 30 13:51:10.450344 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:51:10.450321    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf\": container with ID starting with 8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf not found: ID does not exist" containerID="8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf"
Oct 30 13:51:10.450417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.450358    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf"} err="failed to get container status \"8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf\": rpc error: code = NotFound desc = could not find container \"8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf\": container with ID starting with 8730b2f992beeb137b2466bd678b345daaed4ddecf4ca773224327b56dcd1dcf not found: ID does not exist"
Oct 30 13:51:10.450417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.450376    4993 scope.go:122] "RemoveContainer" containerID="1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd"
Oct 30 13:51:10.450648 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:51:10.450623    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd\": container with ID starting with 1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd not found: ID does not exist" containerID="1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd"
Oct 30 13:51:10.450702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.450658    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd"} err="failed to get container status \"1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd\": rpc error: code = NotFound desc = could not find container \"1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd\": container with ID starting with 1b135a8b96ce0fa55e3ad2b0575b500c7e3818ddcad97bb9e9b5655053c4c1cd not found: ID does not exist"
Oct 30 13:51:10.525328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.525285    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.525328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.525323    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.525565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.525352    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.525565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.525362    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28kz\" (UniqueName: \"kubernetes.io/projected/7b5ff6f0-dce9-4419-aa9e-abe3045212f7-kube-api-access-h28kz\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:51:10.775957 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.775914    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"]
Oct 30 13:51:10.786827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:10.786792    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-tv89c"]
Oct 30 13:51:11.693724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:11.693689    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" path="/var/lib/kubelet/pods/7b5ff6f0-dce9-4419-aa9e-abe3045212f7/volumes"
Oct 30 13:51:23.516898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.516857    4993 scope.go:122] "RemoveContainer" containerID="dadf5f79a462a9103b3e07eaede38382deff533dd8230a398a8d95f354450102"
Oct 30 13:51:23.531242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.531167    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"]
Oct 30 13:51:23.531873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.531846    4993 scope.go:122] "RemoveContainer" containerID="8b6723c97c1bda66aaf7c7a0129ac865206b1c32371bd897bb7d954b78369856"
Oct 30 13:51:23.532063 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532037    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy"
Oct 30 13:51:23.532177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532065    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy"
Oct 30 13:51:23.532177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532088    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy"
Oct 30 13:51:23.532177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532096    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy"
Oct 30 13:51:23.532177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532109    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy"
Oct 30 13:51:23.532177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532116    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy"
Oct 30 13:51:23.532177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532158    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2a86cdcd-ede9-4475-8c24-5518364f7b64" containerName="revoke-keys"
Oct 30 13:51:23.532177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532167    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a86cdcd-ede9-4475-8c24-5518364f7b64" containerName="revoke-keys"
Oct 30 13:51:23.532516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532186    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy"
Oct 30 13:51:23.532516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532193    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy"
Oct 30 13:51:23.532516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532388    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy"
Oct 30 13:51:23.532516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532407    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy"
Oct 30 13:51:23.532516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532419    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a86cdcd-ede9-4475-8c24-5518364f7b64" containerName="revoke-keys"
Oct 30 13:51:23.532516 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.532429    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5ff6f0-dce9-4419-aa9e-abe3045212f7" containerName="istio-proxy"
Oct 30 13:51:23.538278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.538228    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.554727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.554695    4993 scope.go:122] "RemoveContainer" containerID="bd872afb2efb13b391a9bedb75daca4cf3c831a37492d03f187a20a897771bce"
Oct 30 13:51:23.565809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.565764    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f49b40\"/\"rate-test-model-f49b40-kserve-self-signed-certs\""
Oct 30 13:51:23.565992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.565881    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f49b40\"/\"kube-root-ca.crt\""
Oct 30 13:51:23.565992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.565955    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f49b40\"/\"openshift-service-ca.crt\""
Oct 30 13:51:23.566110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.566014    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f49b40\"/\"default-dockercfg-dpxw7\""
Oct 30 13:51:23.574835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.574795    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"]
Oct 30 13:51:23.603079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.603037    4993 scope.go:122] "RemoveContainer" containerID="80a6c15f6f020655eb578f0d5876056b606b3e914d46aa8de8175409b0414e05"
Oct 30 13:51:23.631799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.631765    4993 scope.go:122] "RemoveContainer" containerID="56971980bd0e30b95e6377dae28adb92e4de6d2c047d37d54477efeed6ae0551"
Oct 30 13:51:23.644130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.644097    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjvgt\" (UniqueName: \"kubernetes.io/projected/6059e369-d2ed-4f31-8409-624fe832c897-kube-api-access-xjvgt\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.644130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.644134    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-dshm\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.644567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.644237    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-home\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.644567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.644311    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-tmp-dir\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.644567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.644381    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6059e369-d2ed-4f31-8409-624fe832c897-tls-certs\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.644567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.644407    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-model-cache\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.746910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.746862    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-tmp-dir\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.747127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.746987    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6059e369-d2ed-4f31-8409-624fe832c897-tls-certs\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.747127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.747035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-model-cache\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.747393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.747142    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjvgt\" (UniqueName: \"kubernetes.io/projected/6059e369-d2ed-4f31-8409-624fe832c897-kube-api-access-xjvgt\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.747393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.747169    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-dshm\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.747393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.747344    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-home\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.749469 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.747810    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-home\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.750171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.750146    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-model-cache\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.756268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.748065    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-tmp-dir\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.782229 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.772066    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6059e369-d2ed-4f31-8409-624fe832c897-tls-certs\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.786567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.786527    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-dshm\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.872230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.848065    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjvgt\" (UniqueName: \"kubernetes.io/projected/6059e369-d2ed-4f31-8409-624fe832c897-kube-api-access-xjvgt\") pod \"rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") " pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:23.900917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:23.900641    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:24.367874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:24.367251    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"]
Oct 30 13:51:24.370553 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:51:24.370512    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6059e369_d2ed_4f31_8409_624fe832c897.slice/crio-b9b969a24ad974e557ce3adffd7925825f7c7e0e5e26a163bde05ad8377f2443 WatchSource:0}: Error finding container b9b969a24ad974e557ce3adffd7925825f7c7e0e5e26a163bde05ad8377f2443: Status 404 returned error can't find the container with id b9b969a24ad974e557ce3adffd7925825f7c7e0e5e26a163bde05ad8377f2443
Oct 30 13:51:24.630199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:24.630107    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4" event={"ID":"6059e369-d2ed-4f31-8409-624fe832c897","Type":"ContainerStarted","Data":"f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41"}
Oct 30 13:51:24.630199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:24.630151    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4" event={"ID":"6059e369-d2ed-4f31-8409-624fe832c897","Type":"ContainerStarted","Data":"b9b969a24ad974e557ce3adffd7925825f7c7e0e5e26a163bde05ad8377f2443"}
Oct 30 13:51:24.711610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:24.711540    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4" podStartSLOduration=1.711519455 podStartE2EDuration="1.711519455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:24.708266314 +0000 UTC m=+4749.183985925" watchObservedRunningTime="2026-10-30 13:51:24.711519455 +0000 UTC m=+4749.187239066"
Oct 30 13:51:24.901850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:24.901756    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:33.901280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:33.901236    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:33.914724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:33.914688    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:34.725221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:34.725174    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:51:38.578545 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:51:38.578501    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:51:50.228238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:50.225954    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"]
Oct 30 13:51:50.229568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:50.229536    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bd2aaf-wtdmd"]
Oct 30 13:51:51.692608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:51.692568    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a86cdcd-ede9-4475-8c24-5518364f7b64" path="/var/lib/kubelet/pods/2a86cdcd-ede9-4475-8c24-5518364f7b64/volumes"
Oct 30 13:51:58.107847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.107805    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"]
Oct 30 13:51:58.108502 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.108484    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6264a5b2-8654-48bc-a5e8-9d26c7e6acd6" containerName="istio-proxy"
Oct 30 13:51:58.111744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.111722    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.127997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.127963    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:51:58.139767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.139730    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:51:58.139952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.139844    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:51:58.139952 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.139886    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:51:58.145371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.145329    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"]
Oct 30 13:51:58.219594 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.219549    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.220485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.220451    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.220765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.220726    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vjlb\" (UniqueName: \"kubernetes.io/projected/a4c3e080-b019-40db-afe8-4792bcbdc6fa-kube-api-access-6vjlb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.322804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.322758    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.323044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.322814    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vjlb\" (UniqueName: \"kubernetes.io/projected/a4c3e080-b019-40db-afe8-4792bcbdc6fa-kube-api-access-6vjlb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.323044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.322865    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.323802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.323772    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.325660 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.325627    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.347251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.347161    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vjlb\" (UniqueName: \"kubernetes.io/projected/a4c3e080-b019-40db-afe8-4792bcbdc6fa-kube-api-access-6vjlb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.427763 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.427663    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:51:58.787026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:58.781861    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"]
Oct 30 13:51:58.795281 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:51:58.793029    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c3e080_b019_40db_afe8_4792bcbdc6fa.slice/crio-442b1a3e7d9b62e4e14e13e8d30cf2934e152ea07479649b596844fae696de07 WatchSource:0}: Error finding container 442b1a3e7d9b62e4e14e13e8d30cf2934e152ea07479649b596844fae696de07: Status 404 returned error can't find the container with id 442b1a3e7d9b62e4e14e13e8d30cf2934e152ea07479649b596844fae696de07
Oct 30 13:51:59.042908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:51:59.042812    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq" event={"ID":"a4c3e080-b019-40db-afe8-4792bcbdc6fa","Type":"ContainerStarted","Data":"442b1a3e7d9b62e4e14e13e8d30cf2934e152ea07479649b596844fae696de07"}
Oct 30 13:52:00.055703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:00.055654    4993 generic.go:356] "Generic (PLEG): container finished" podID="a4c3e080-b019-40db-afe8-4792bcbdc6fa" containerID="97b00bab8779753009000e09b9dd58b26217bbaaf11b7fe8f5e5f87e6b571821" exitCode=0
Oct 30 13:52:00.056148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:00.055714    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq" event={"ID":"a4c3e080-b019-40db-afe8-4792bcbdc6fa","Type":"ContainerDied","Data":"97b00bab8779753009000e09b9dd58b26217bbaaf11b7fe8f5e5f87e6b571821"}
Oct 30 13:52:01.269567 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.269533    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:52:01.311113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.311066    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-xks-ca\") pod \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") "
Oct 30 13:52:01.311375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.311163    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c3e080-b019-40db-afe8-4792bcbdc6fa-kube-api-access-6vjlb\" (UniqueName: \"kubernetes.io/projected/a4c3e080-b019-40db-afe8-4792bcbdc6fa-kube-api-access-6vjlb\") pod \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") "
Oct 30 13:52:01.311375 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.311229    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-openshift-ca\") pod \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\" (UID: \"a4c3e080-b019-40db-afe8-4792bcbdc6fa\") "
Oct 30 13:52:01.312269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.312239    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-openshift-ca" pod "a4c3e080-b019-40db-afe8-4792bcbdc6fa" (UID: "a4c3e080-b019-40db-afe8-4792bcbdc6fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:01.314475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.314443    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c3e080-b019-40db-afe8-4792bcbdc6fa-kube-api-access-6vjlb" pod "a4c3e080-b019-40db-afe8-4792bcbdc6fa" (UID: "a4c3e080-b019-40db-afe8-4792bcbdc6fa"). InnerVolumeSpecName "kube-api-access-6vjlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:01.317668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.317640    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-xks-ca" pod "a4c3e080-b019-40db-afe8-4792bcbdc6fa" (UID: "a4c3e080-b019-40db-afe8-4792bcbdc6fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:01.413219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.413099    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:01.413219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.413135    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vjlb\" (UniqueName: \"kubernetes.io/projected/a4c3e080-b019-40db-afe8-4792bcbdc6fa-kube-api-access-6vjlb\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:01.413219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:01.413146    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c3e080-b019-40db-afe8-4792bcbdc6fa-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.091282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:02.091236    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq" event={"ID":"a4c3e080-b019-40db-afe8-4792bcbdc6fa","Type":"ContainerDied","Data":"442b1a3e7d9b62e4e14e13e8d30cf2934e152ea07479649b596844fae696de07"}
Oct 30 13:52:02.091484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:02.091310    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442b1a3e7d9b62e4e14e13e8d30cf2934e152ea07479649b596844fae696de07"
Oct 30 13:52:02.091484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:02.091250    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"
Oct 30 13:52:05.993770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:05.993725    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"]
Oct 30 13:52:05.994609 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:05.994586    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a4c3e080-b019-40db-afe8-4792bcbdc6fa" containerName="revoke-keys"
Oct 30 13:52:05.994683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:05.994613    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c3e080-b019-40db-afe8-4792bcbdc6fa" containerName="revoke-keys"
Oct 30 13:52:05.994832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:05.994814    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c3e080-b019-40db-afe8-4792bcbdc6fa" containerName="revoke-keys"
Oct 30 13:52:05.998615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:05.998584    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.032004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.031950    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"]
Oct 30 13:52:06.038621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.038565    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:52:06.038868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.038615    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:52:06.087305 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.087257    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.087532 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.087399    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cxd\" (UniqueName: \"kubernetes.io/projected/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-kube-api-access-h5cxd\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.087632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.087613    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.109562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.109520    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:52:06.109562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.109548    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:52:06.188725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.188685    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.188936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.188813    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.188936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.188879    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cxd\" (UniqueName: \"kubernetes.io/projected/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-kube-api-access-h5cxd\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.189497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.189459    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.191893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.191854    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.213060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.213014    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cxd\" (UniqueName: \"kubernetes.io/projected/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-kube-api-access-h5cxd\") pod \"maas-api-revoke-sub-bad16cd7-jp9t2\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.318712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.318618    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:06.632016 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:06.631980    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"]
Oct 30 13:52:07.030393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:07.030349    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"]
Oct 30 13:52:07.042457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:07.042418    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c495032a-rlccd"]
Oct 30 13:52:07.145406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:07.145295    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2" event={"ID":"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac","Type":"ContainerStarted","Data":"b018a380f05e726f76fedd6252c37283d62bd45bcd5c00518a2bff831e2e0f06"}
Oct 30 13:52:07.145406 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:07.145357    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2" event={"ID":"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac","Type":"ContainerStarted","Data":"c3cd04fe8c630d6c97f245c1c19c71035959cd506ba9569a9bb12a777029237d"}
Oct 30 13:52:07.185744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:07.185669    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2" podStartSLOduration=2.185648586 podStartE2EDuration="2.185648586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:07.184845567 +0000 UTC m=+4791.660565177" watchObservedRunningTime="2026-10-30 13:52:07.185648586 +0000 UTC m=+4791.661368195"
Oct 30 13:52:07.694492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:07.694455    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786c1728-4779-43d9-b7a6-b13c50639968" path="/var/lib/kubelet/pods/786c1728-4779-43d9-b7a6-b13c50639968/volumes"
Oct 30 13:52:08.155661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:08.155621    4993 generic.go:356] "Generic (PLEG): container finished" podID="2cfa9ee8-bb60-4c72-af4a-8c4e48084bac" containerID="b018a380f05e726f76fedd6252c37283d62bd45bcd5c00518a2bff831e2e0f06" exitCode=0
Oct 30 13:52:08.155661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:08.155666    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2" event={"ID":"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac","Type":"ContainerDied","Data":"b018a380f05e726f76fedd6252c37283d62bd45bcd5c00518a2bff831e2e0f06"}
Oct 30 13:52:08.570770 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:52:08.570729    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:52:09.488907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.488874    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:09.576156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.576101    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-xks-ca\") pod \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") "
Oct 30 13:52:09.576360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.576317    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-openshift-ca\") pod \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") "
Oct 30 13:52:09.576416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.576369    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-kube-api-access-h5cxd\" (UniqueName: \"kubernetes.io/projected/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-kube-api-access-h5cxd\") pod \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\" (UID: \"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac\") "
Oct 30 13:52:09.576715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.576684    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-openshift-ca" pod "2cfa9ee8-bb60-4c72-af4a-8c4e48084bac" (UID: "2cfa9ee8-bb60-4c72-af4a-8c4e48084bac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:09.576972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.576953    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:09.578683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.578659    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-kube-api-access-h5cxd" pod "2cfa9ee8-bb60-4c72-af4a-8c4e48084bac" (UID: "2cfa9ee8-bb60-4c72-af4a-8c4e48084bac"). InnerVolumeSpecName "kube-api-access-h5cxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:09.578939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.578911    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-xks-ca" pod "2cfa9ee8-bb60-4c72-af4a-8c4e48084bac" (UID: "2cfa9ee8-bb60-4c72-af4a-8c4e48084bac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:09.678181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.678139    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:09.678181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:09.678175    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5cxd\" (UniqueName: \"kubernetes.io/projected/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac-kube-api-access-h5cxd\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:10.185866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:10.185799    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"
Oct 30 13:52:10.185866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:10.185814    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2" event={"ID":"2cfa9ee8-bb60-4c72-af4a-8c4e48084bac","Type":"ContainerDied","Data":"c3cd04fe8c630d6c97f245c1c19c71035959cd506ba9569a9bb12a777029237d"}
Oct 30 13:52:10.186155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:10.185940    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3cd04fe8c630d6c97f245c1c19c71035959cd506ba9569a9bb12a777029237d"
Oct 30 13:52:16.382014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.381965    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"]
Oct 30 13:52:16.382808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.382781    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2cfa9ee8-bb60-4c72-af4a-8c4e48084bac" containerName="revoke-keys"
Oct 30 13:52:16.382904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.382810    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cfa9ee8-bb60-4c72-af4a-8c4e48084bac" containerName="revoke-keys"
Oct 30 13:52:16.383019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.383004    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cfa9ee8-bb60-4c72-af4a-8c4e48084bac" containerName="revoke-keys"
Oct 30 13:52:16.387192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.387163    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.401743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.401699    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:52:16.413860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.413817    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:52:16.413860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.413818    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:52:16.414087 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.413820    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:52:16.419180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.419138    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"]
Oct 30 13:52:16.490048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.490005    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.490304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.490153    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvmm\" (UniqueName: \"kubernetes.io/projected/0a583310-a797-4086-b44d-a085a01abd9a-kube-api-access-njvmm\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.490304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.490264    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.591283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.591201    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.591506 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.591351    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvmm\" (UniqueName: \"kubernetes.io/projected/0a583310-a797-4086-b44d-a085a01abd9a-kube-api-access-njvmm\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.591506 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.591449    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.592199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.592171    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.594281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.594253    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.600991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.600966    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:52:16.615157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.615125    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvmm\" (UniqueName: \"kubernetes.io/projected/0a583310-a797-4086-b44d-a085a01abd9a-kube-api-access-njvmm\") pod \"maas-api-revoke-sub-2f7e4664-jfndt\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:16.645507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.645423    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:52:16.702035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:16.701984    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:17.029110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:17.029063    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"]
Oct 30 13:52:17.030098 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:52:17.030061    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a583310_a797_4086_b44d_a085a01abd9a.slice/crio-4db6733830b00a6ae001401e54cae64ff8b8c039ce5f650253becf9aa912bec0 WatchSource:0}: Error finding container 4db6733830b00a6ae001401e54cae64ff8b8c039ce5f650253becf9aa912bec0: Status 404 returned error can't find the container with id 4db6733830b00a6ae001401e54cae64ff8b8c039ce5f650253becf9aa912bec0
Oct 30 13:52:17.246815 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:17.246774    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt" event={"ID":"0a583310-a797-4086-b44d-a085a01abd9a","Type":"ContainerStarted","Data":"4db6733830b00a6ae001401e54cae64ff8b8c039ce5f650253becf9aa912bec0"}
Oct 30 13:52:18.258360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:18.258316    4993 generic.go:356] "Generic (PLEG): container finished" podID="0a583310-a797-4086-b44d-a085a01abd9a" containerID="f1c271486b76da2b64cde1198541a5c6d8bd27ce38a2544633a1e86a360af25f" exitCode=0
Oct 30 13:52:18.258853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:18.258405    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt" event={"ID":"0a583310-a797-4086-b44d-a085a01abd9a","Type":"ContainerDied","Data":"f1c271486b76da2b64cde1198541a5c6d8bd27ce38a2544633a1e86a360af25f"}
Oct 30 13:52:19.529130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.529053    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:19.652971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.652922    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a583310-a797-4086-b44d-a085a01abd9a-kube-api-access-njvmm\" (UniqueName: \"kubernetes.io/projected/0a583310-a797-4086-b44d-a085a01abd9a-kube-api-access-njvmm\") pod \"0a583310-a797-4086-b44d-a085a01abd9a\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") "
Oct 30 13:52:19.653263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.653097    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-openshift-ca\") pod \"0a583310-a797-4086-b44d-a085a01abd9a\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") "
Oct 30 13:52:19.653263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.653162    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-xks-ca\") pod \"0a583310-a797-4086-b44d-a085a01abd9a\" (UID: \"0a583310-a797-4086-b44d-a085a01abd9a\") "
Oct 30 13:52:19.653682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.653659    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-openshift-ca" pod "0a583310-a797-4086-b44d-a085a01abd9a" (UID: "0a583310-a797-4086-b44d-a085a01abd9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:19.653857 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.653830    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:19.655463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.655434    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a583310-a797-4086-b44d-a085a01abd9a-kube-api-access-njvmm" pod "0a583310-a797-4086-b44d-a085a01abd9a" (UID: "0a583310-a797-4086-b44d-a085a01abd9a"). InnerVolumeSpecName "kube-api-access-njvmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:19.655593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.655516    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-xks-ca" pod "0a583310-a797-4086-b44d-a085a01abd9a" (UID: "0a583310-a797-4086-b44d-a085a01abd9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:19.755333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.755290    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583310-a797-4086-b44d-a085a01abd9a-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:19.755333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:19.755329    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njvmm\" (UniqueName: \"kubernetes.io/projected/0a583310-a797-4086-b44d-a085a01abd9a-kube-api-access-njvmm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:20.278575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:20.278540    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"
Oct 30 13:52:20.278770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:20.278585    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt" event={"ID":"0a583310-a797-4086-b44d-a085a01abd9a","Type":"ContainerDied","Data":"4db6733830b00a6ae001401e54cae64ff8b8c039ce5f650253becf9aa912bec0"}
Oct 30 13:52:20.278770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:20.278651    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db6733830b00a6ae001401e54cae64ff8b8c039ce5f650253becf9aa912bec0"
Oct 30 13:52:23.986011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:23.985969    4993 scope.go:122] "RemoveContainer" containerID="50b407247f68315930ca8184268a4e454ae2a2fb1f3b7bf01b8a992169aad713"
Oct 30 13:52:23.997350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:23.997323    4993 scope.go:122] "RemoveContainer" containerID="13a919a357d6886d8af4c9acc133c0b61a91ee84db8672385ffc65ed64b945ae"
Oct 30 13:52:24.007986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:24.007958    4993 scope.go:122] "RemoveContainer" containerID="949b2eaad0bfb3010de2bfd7eab089151dc5bfc202adf388d4f52eb49418ff2f"
Oct 30 13:52:24.018340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:24.018316    4993 scope.go:122] "RemoveContainer" containerID="c79fef8bb2207b3aadb1e53dc5c403528cbc60be6af66fa54da90fd798fb0465"
Oct 30 13:52:24.028348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:24.028324    4993 scope.go:122] "RemoveContainer" containerID="5fc1b260bacf1f97e2451aa4c69539c2c9fcfd0fbbfac5560f1f730e666b93b3"
Oct 30 13:52:27.229104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.229054    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"]
Oct 30 13:52:27.229748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.229716    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0a583310-a797-4086-b44d-a085a01abd9a" containerName="revoke-keys"
Oct 30 13:52:27.229748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.229740    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a583310-a797-4086-b44d-a085a01abd9a" containerName="revoke-keys"
Oct 30 13:52:27.229970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.229889    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a583310-a797-4086-b44d-a085a01abd9a" containerName="revoke-keys"
Oct 30 13:52:27.233294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.233266    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.253534 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.253496    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"]
Oct 30 13:52:27.260690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.260655    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:52:27.260690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.260641    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:52:27.260690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.260692    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:52:27.260986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.260757    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:52:27.285761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.285700    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.285936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.285851    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtq4\" (UniqueName: \"kubernetes.io/projected/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-kube-api-access-nrtq4\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.285936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.285917    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.387061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.387012    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtq4\" (UniqueName: \"kubernetes.io/projected/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-kube-api-access-nrtq4\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.387315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.387129    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.387477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.387374    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.387949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.387919    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.389984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.389955    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.410477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.410440    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtq4\" (UniqueName: \"kubernetes.io/projected/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-kube-api-access-nrtq4\") pod \"maas-api-revoke-sub-dd7760d8-8krch\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.547995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.547888    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:27.868485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:27.868446    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"]
Oct 30 13:52:27.870024 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:52:27.869986    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051ad4fd_bf88_4f71_93d5_acc9b9c5c5a0.slice/crio-0d76a214e03e64979da1a33898d33ec6fc2435f8ac7010a55e9ae380375ce772 WatchSource:0}: Error finding container 0d76a214e03e64979da1a33898d33ec6fc2435f8ac7010a55e9ae380375ce772: Status 404 returned error can't find the container with id 0d76a214e03e64979da1a33898d33ec6fc2435f8ac7010a55e9ae380375ce772
Oct 30 13:52:28.349513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:28.349463    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch" event={"ID":"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0","Type":"ContainerStarted","Data":"7b427ba1f0bd82b21e9082e6ea5517a0d043c63a6dc66301d0c251a31108f35a"}
Oct 30 13:52:28.349948 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:28.349525    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch" event={"ID":"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0","Type":"ContainerStarted","Data":"0d76a214e03e64979da1a33898d33ec6fc2435f8ac7010a55e9ae380375ce772"}
Oct 30 13:52:28.393002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:28.392920    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch" podStartSLOduration=1.392899123 podStartE2EDuration="1.392899123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:28.390635574 +0000 UTC m=+4812.866355184" watchObservedRunningTime="2026-10-30 13:52:28.392899123 +0000 UTC m=+4812.868618734"
Oct 30 13:52:29.361688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:29.361647    4993 generic.go:356] "Generic (PLEG): container finished" podID="051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0" containerID="7b427ba1f0bd82b21e9082e6ea5517a0d043c63a6dc66301d0c251a31108f35a" exitCode=0
Oct 30 13:52:29.362128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:29.361748    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch" event={"ID":"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0","Type":"ContainerDied","Data":"7b427ba1f0bd82b21e9082e6ea5517a0d043c63a6dc66301d0c251a31108f35a"}
Oct 30 13:52:30.545986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.545958    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:30.580956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.580916    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-kube-api-access-nrtq4\" (UniqueName: \"kubernetes.io/projected/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-kube-api-access-nrtq4\") pod \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") "
Oct 30 13:52:30.581185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.581165    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-openshift-ca\") pod \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") "
Oct 30 13:52:30.581284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.581194    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-xks-ca\") pod \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\" (UID: \"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0\") "
Oct 30 13:52:30.581720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.581656    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-openshift-ca" pod "051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0" (UID: "051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:30.581867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.581837    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:30.583749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.583722    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-xks-ca" pod "051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0" (UID: "051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:30.583868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.583761    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-kube-api-access-nrtq4" pod "051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0" (UID: "051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0"). InnerVolumeSpecName "kube-api-access-nrtq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:30.683094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.682996    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtq4\" (UniqueName: \"kubernetes.io/projected/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-kube-api-access-nrtq4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:30.683094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:30.683044    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:31.383275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:31.383191    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch" event={"ID":"051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0","Type":"ContainerDied","Data":"0d76a214e03e64979da1a33898d33ec6fc2435f8ac7010a55e9ae380375ce772"}
Oct 30 13:52:31.383275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:31.383261    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"
Oct 30 13:52:31.383572 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:31.383327    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d76a214e03e64979da1a33898d33ec6fc2435f8ac7010a55e9ae380375ce772"
Oct 30 13:52:37.854614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.854565    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"]
Oct 30 13:52:37.855348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.855192    4993 cpu_manager.go:414] "containerMap: removing container" podUID="051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0" containerName="revoke-keys"
Oct 30 13:52:37.855348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.855223    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0" containerName="revoke-keys"
Oct 30 13:52:37.855488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.855362    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0" containerName="revoke-keys"
Oct 30 13:52:37.858705 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.858678    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:37.885154 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.885115    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:52:37.885485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.885404    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:52:37.885485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.885432    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:52:37.885485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.885522    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:52:37.890116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:37.890075    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"]
Oct 30 13:52:38.030985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.030940    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.031281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.031259    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2dk\" (UniqueName: \"kubernetes.io/projected/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-kube-api-access-lk2dk\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.031342 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.031311    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.140292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.135405    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2dk\" (UniqueName: \"kubernetes.io/projected/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-kube-api-access-lk2dk\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.140292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.135496    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.140292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.135594    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.140292 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.136610    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.147690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.141651    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.160741 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.160702    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2dk\" (UniqueName: \"kubernetes.io/projected/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-kube-api-access-lk2dk\") pod \"maas-api-revoke-sub-bd48803d-7cxxs\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.177300 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.177261    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:38.494012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:38.493981    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"]
Oct 30 13:52:38.496095 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:52:38.496046    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae167aa9_d7d6_4a75_a211_c16a83b3e0e2.slice/crio-41f0feba601fc0b7c0bf457dcc60509a996b1236f2d472f1399344232be70881 WatchSource:0}: Error finding container 41f0feba601fc0b7c0bf457dcc60509a996b1236f2d472f1399344232be70881: Status 404 returned error can't find the container with id 41f0feba601fc0b7c0bf457dcc60509a996b1236f2d472f1399344232be70881
Oct 30 13:52:38.574660 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:52:38.574613    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:52:39.456426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:39.456383    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs" event={"ID":"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2","Type":"ContainerStarted","Data":"ff309e9419b6b15f9e5d17b0a5e5e0176ca342c396880428ca8491d21cb39691"}
Oct 30 13:52:39.457116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:39.456448    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs" event={"ID":"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2","Type":"ContainerStarted","Data":"41f0feba601fc0b7c0bf457dcc60509a996b1236f2d472f1399344232be70881"}
Oct 30 13:52:39.498471 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:39.498417    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs" podStartSLOduration=2.498401557 podStartE2EDuration="2.498401557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:39.496798652 +0000 UTC m=+4823.972518259" watchObservedRunningTime="2026-10-30 13:52:39.498401557 +0000 UTC m=+4823.974121166"
Oct 30 13:52:40.088284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:40.088217    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"]
Oct 30 13:52:40.094671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:40.094631    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71fbeb-8pxxq"]
Oct 30 13:52:40.467179 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:40.467080    4993 generic.go:356] "Generic (PLEG): container finished" podID="ae167aa9-d7d6-4a75-a211-c16a83b3e0e2" containerID="ff309e9419b6b15f9e5d17b0a5e5e0176ca342c396880428ca8491d21cb39691" exitCode=0
Oct 30 13:52:40.467658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:40.467162    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs" event={"ID":"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2","Type":"ContainerDied","Data":"ff309e9419b6b15f9e5d17b0a5e5e0176ca342c396880428ca8491d21cb39691"}
Oct 30 13:52:41.653378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.653348    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:41.692270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.692233    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c3e080-b019-40db-afe8-4792bcbdc6fa" path="/var/lib/kubelet/pods/a4c3e080-b019-40db-afe8-4792bcbdc6fa/volumes"
Oct 30 13:52:41.785621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.785588    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-openshift-ca\") pod \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") "
Oct 30 13:52:41.785826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.785676    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-kube-api-access-lk2dk\" (UniqueName: \"kubernetes.io/projected/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-kube-api-access-lk2dk\") pod \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") "
Oct 30 13:52:41.785826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.785783    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-xks-ca\") pod \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\" (UID: \"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2\") "
Oct 30 13:52:41.786088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.786030    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-openshift-ca" pod "ae167aa9-d7d6-4a75-a211-c16a83b3e0e2" (UID: "ae167aa9-d7d6-4a75-a211-c16a83b3e0e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:41.786771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.786732    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:41.788654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.788625    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-kube-api-access-lk2dk" pod "ae167aa9-d7d6-4a75-a211-c16a83b3e0e2" (UID: "ae167aa9-d7d6-4a75-a211-c16a83b3e0e2"). InnerVolumeSpecName "kube-api-access-lk2dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:41.788930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.788903    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-xks-ca" pod "ae167aa9-d7d6-4a75-a211-c16a83b3e0e2" (UID: "ae167aa9-d7d6-4a75-a211-c16a83b3e0e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:41.888457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.888402    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2dk\" (UniqueName: \"kubernetes.io/projected/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-kube-api-access-lk2dk\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:41.888457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:41.888448    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:42.485176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:42.485141    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"
Oct 30 13:52:42.485426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:42.485180    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs" event={"ID":"ae167aa9-d7d6-4a75-a211-c16a83b3e0e2","Type":"ContainerDied","Data":"41f0feba601fc0b7c0bf457dcc60509a996b1236f2d472f1399344232be70881"}
Oct 30 13:52:42.485426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:42.485275    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f0feba601fc0b7c0bf457dcc60509a996b1236f2d472f1399344232be70881"
Oct 30 13:52:47.921665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:47.921623    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"]
Oct 30 13:52:47.922244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:47.921891    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy" containerID="cri-o://fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b" gracePeriod=30
Oct 30 13:52:48.625752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.625713    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"]
Oct 30 13:52:48.626303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.626281    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ae167aa9-d7d6-4a75-a211-c16a83b3e0e2" containerName="revoke-keys"
Oct 30 13:52:48.626303 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.626296    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae167aa9-d7d6-4a75-a211-c16a83b3e0e2" containerName="revoke-keys"
Oct 30 13:52:48.626457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.626427    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae167aa9-d7d6-4a75-a211-c16a83b3e0e2" containerName="revoke-keys"
Oct 30 13:52:48.631082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.631062    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.652861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.652348    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:52:48.686330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.678070    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:52:48.715142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.715106    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:52:48.744555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.716651    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:52:48.744920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.741499    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.745661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.745636    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.746042 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.746026    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qss6w\" (UniqueName: \"kubernetes.io/projected/0b9c4cc0-5614-4656-b93f-d93cce54da32-kube-api-access-qss6w\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.773243 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.770229    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"]
Oct 30 13:52:48.807368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.801570    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"]
Oct 30 13:52:48.848232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.847306    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.848232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.847454    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.848232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.847651    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qss6w\" (UniqueName: \"kubernetes.io/projected/0b9c4cc0-5614-4656-b93f-d93cce54da32-kube-api-access-qss6w\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.874234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.848805    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.951283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.944480    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.973825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.973752    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qss6w\" (UniqueName: \"kubernetes.io/projected/0b9c4cc0-5614-4656-b93f-d93cce54da32-kube-api-access-qss6w\") pod \"maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:48.980732 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:48.980443    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:49.362850 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:49.362799    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:52:49.363067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:49.362874    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:52:49.367965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:49.367932    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"]
Oct 30 13:52:49.370588 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:52:49.370556    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9c4cc0_5614_4656_b93f_d93cce54da32.slice/crio-ff82a8a7cb5085fb42bde03bf79eb3f6fd127e1f7ca51909cdc9095fa28fff67 WatchSource:0}: Error finding container ff82a8a7cb5085fb42bde03bf79eb3f6fd127e1f7ca51909cdc9095fa28fff67: Status 404 returned error can't find the container with id ff82a8a7cb5085fb42bde03bf79eb3f6fd127e1f7ca51909cdc9095fa28fff67
Oct 30 13:52:49.585134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:49.585093    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72" event={"ID":"0b9c4cc0-5614-4656-b93f-d93cce54da32","Type":"ContainerStarted","Data":"ff82a8a7cb5085fb42bde03bf79eb3f6fd127e1f7ca51909cdc9095fa28fff67"}
Oct 30 13:52:50.595897 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:50.595851    4993 generic.go:356] "Generic (PLEG): container finished" podID="0b9c4cc0-5614-4656-b93f-d93cce54da32" containerID="98c322d62bc04c182145489be7c4d41ec2f78559acd11b48ce46161c0c1a072b" exitCode=0
Oct 30 13:52:50.596400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:50.595900    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72" event={"ID":"0b9c4cc0-5614-4656-b93f-d93cce54da32","Type":"ContainerDied","Data":"98c322d62bc04c182145489be7c4d41ec2f78559acd11b48ce46161c0c1a072b"}
Oct 30 13:52:51.793119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:51.793083    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:51.903247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:51.903136    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9c4cc0-5614-4656-b93f-d93cce54da32-kube-api-access-qss6w\" (UniqueName: \"kubernetes.io/projected/0b9c4cc0-5614-4656-b93f-d93cce54da32-kube-api-access-qss6w\") pod \"0b9c4cc0-5614-4656-b93f-d93cce54da32\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") "
Oct 30 13:52:51.903247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:51.903238    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-xks-ca\") pod \"0b9c4cc0-5614-4656-b93f-d93cce54da32\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") "
Oct 30 13:52:51.903468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:51.903366    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-openshift-ca\") pod \"0b9c4cc0-5614-4656-b93f-d93cce54da32\" (UID: \"0b9c4cc0-5614-4656-b93f-d93cce54da32\") "
Oct 30 13:52:51.903756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:51.903727    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-openshift-ca" pod "0b9c4cc0-5614-4656-b93f-d93cce54da32" (UID: "0b9c4cc0-5614-4656-b93f-d93cce54da32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:51.903966 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:51.903946    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:51.905687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:51.905654    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-xks-ca" pod "0b9c4cc0-5614-4656-b93f-d93cce54da32" (UID: "0b9c4cc0-5614-4656-b93f-d93cce54da32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:51.905798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:51.905690    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9c4cc0-5614-4656-b93f-d93cce54da32-kube-api-access-qss6w" pod "0b9c4cc0-5614-4656-b93f-d93cce54da32" (UID: "0b9c4cc0-5614-4656-b93f-d93cce54da32"). InnerVolumeSpecName "kube-api-access-qss6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:52.004903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:52.004859    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qss6w\" (UniqueName: \"kubernetes.io/projected/0b9c4cc0-5614-4656-b93f-d93cce54da32-kube-api-access-qss6w\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:52.004903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:52.004895    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9c4cc0-5614-4656-b93f-d93cce54da32-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:52.613131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:52.613100    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"
Oct 30 13:52:52.613131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:52.613125    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72" event={"ID":"0b9c4cc0-5614-4656-b93f-d93cce54da32","Type":"ContainerDied","Data":"ff82a8a7cb5085fb42bde03bf79eb3f6fd127e1f7ca51909cdc9095fa28fff67"}
Oct 30 13:52:52.613422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:52.613191    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff82a8a7cb5085fb42bde03bf79eb3f6fd127e1f7ca51909cdc9095fa28fff67"
Oct 30 13:52:53.225317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.225290    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:52:53.326813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.326775    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-socket\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.326813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.326822    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-credential-socket\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.327092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.326882    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-envoy\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.327092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.326994    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-istio-token\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-istio-token\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.327092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327040    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-certs\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.327092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327065    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-kube-api-access-f64qq\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-kube-api-access-f64qq\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.327356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327136    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-data\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.327356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327163    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79100f5f-7848-488a-819a-eb9227e0e055-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79100f5f-7848-488a-819a-eb9227e0e055-istiod-ca-cert\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.327356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327158    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-socket" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:53.327356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327187    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79100f5f-7848-488a-819a-eb9227e0e055-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79100f5f-7848-488a-819a-eb9227e0e055-istio-podinfo\") pod \"79100f5f-7848-488a-819a-eb9227e0e055\" (UID: \"79100f5f-7848-488a-819a-eb9227e0e055\") "
Oct 30 13:52:53.327356 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327255    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-credential-socket" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:53.327651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327416    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-certs" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:53.327651 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327511    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-data" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:53.327819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327773    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.327819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327796    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.327819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327809    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.327995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327824    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.327995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.327967    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79100f5f-7848-488a-819a-eb9227e0e055-istiod-ca-cert" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:53.329858 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.329831    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79100f5f-7848-488a-819a-eb9227e0e055-istio-podinfo" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:52:53.330486 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.330460    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-envoy" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:53.330743 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.330716    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-kube-api-access-f64qq" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "kube-api-access-f64qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:53.330853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.330786    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-istio-token" pod "79100f5f-7848-488a-819a-eb9227e0e055" (UID: "79100f5f-7848-488a-819a-eb9227e0e055"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:53.428861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.428823    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.428861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.428857    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f64qq\" (UniqueName: \"kubernetes.io/projected/79100f5f-7848-488a-819a-eb9227e0e055-kube-api-access-f64qq\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.428861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.428868    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79100f5f-7848-488a-819a-eb9227e0e055-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.429337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.428882    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79100f5f-7848-488a-819a-eb9227e0e055-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.429337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.428894    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79100f5f-7848-488a-819a-eb9227e0e055-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:53.622374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.622340    4993 generic.go:356] "Generic (PLEG): container finished" podID="79100f5f-7848-488a-819a-eb9227e0e055" containerID="fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b" exitCode=0
Oct 30 13:52:53.622564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.622394    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"
Oct 30 13:52:53.622564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.622398    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" event={"ID":"79100f5f-7848-488a-819a-eb9227e0e055","Type":"ContainerDied","Data":"fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b"}
Oct 30 13:52:53.622564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.622525    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv" event={"ID":"79100f5f-7848-488a-819a-eb9227e0e055","Type":"ContainerDied","Data":"4f47ae1dd9d1c11add9ca37681052d7ba8e794f5e3669a07b032bb28b00bc8ab"}
Oct 30 13:52:53.622564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.622547    4993 scope.go:122] "RemoveContainer" containerID="fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b"
Oct 30 13:52:53.636317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.636290    4993 scope.go:122] "RemoveContainer" containerID="6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f"
Oct 30 13:52:53.649367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.649339    4993 scope.go:122] "RemoveContainer" containerID="fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b"
Oct 30 13:52:53.649719 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:52:53.649687    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b\": container with ID starting with fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b not found: ID does not exist" containerID="fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b"
Oct 30 13:52:53.649796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.649728    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b"} err="failed to get container status \"fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b\": rpc error: code = NotFound desc = could not find container \"fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b\": container with ID starting with fb9d361291c9d22ec54d12b647b4a64bd0a141e85bf8b75a5308ba48cedf2a1b not found: ID does not exist"
Oct 30 13:52:53.649796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.649748    4993 scope.go:122] "RemoveContainer" containerID="6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f"
Oct 30 13:52:53.649985 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:52:53.649965    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f\": container with ID starting with 6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f not found: ID does not exist" containerID="6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f"
Oct 30 13:52:53.650028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.649990    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f"} err="failed to get container status \"6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f\": rpc error: code = NotFound desc = could not find container \"6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f\": container with ID starting with 6fefe30c274bf170f6c83049d4ac31ec891ee0dad359c3b6aee4984adbea631f not found: ID does not exist"
Oct 30 13:52:53.705454 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.705419    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"]
Oct 30 13:52:53.705454 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:53.705461    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71fbeb-openshift-default-688f4d9669-7f6lv"]
Oct 30 13:52:54.004845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.004807    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"]
Oct 30 13:52:54.247126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.244371    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"]
Oct 30 13:52:54.247126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.245143    4993 cpu_manager.go:414] "containerMap: removing container" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy"
Oct 30 13:52:54.247126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.245163    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy"
Oct 30 13:52:54.247126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.245201    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0b9c4cc0-5614-4656-b93f-d93cce54da32" containerName="revoke-keys"
Oct 30 13:52:54.247126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.245226    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9c4cc0-5614-4656-b93f-d93cce54da32" containerName="revoke-keys"
Oct 30 13:52:54.247126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.245368    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9c4cc0-5614-4656-b93f-d93cce54da32" containerName="revoke-keys"
Oct 30 13:52:54.247126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.245381    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy"
Oct 30 13:52:54.252153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.252116    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.276803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.276369    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:52:54.276803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.276477    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:52:54.276803 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.276565    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:52:54.286190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.283412    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:52:54.323777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.314843    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"]
Oct 30 13:52:54.346182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.345899    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.346182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.346070    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q822r\" (UniqueName: \"kubernetes.io/projected/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-kube-api-access-q822r\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.346497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.346266    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.447281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.447236    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.447490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.447320    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.447490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.447382    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q822r\" (UniqueName: \"kubernetes.io/projected/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-kube-api-access-q822r\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.448006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.447978    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.449857 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.449833    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.471089 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.471053    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q822r\" (UniqueName: \"kubernetes.io/projected/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-kube-api-access-q822r\") pod \"maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.628250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.628156    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:54.938503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:54.938462    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"]
Oct 30 13:52:54.939358 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:52:54.939330    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288c920e_ace3_4b8a_b5fd_2c94dfb4f056.slice/crio-1ba1ff56beb6dd80909dd59a2af3d6e7dbf110cd35c4a854cda8859b7f446a63 WatchSource:0}: Error finding container 1ba1ff56beb6dd80909dd59a2af3d6e7dbf110cd35c4a854cda8859b7f446a63: Status 404 returned error can't find the container with id 1ba1ff56beb6dd80909dd59a2af3d6e7dbf110cd35c4a854cda8859b7f446a63
Oct 30 13:52:55.641895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:55.641786    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6" event={"ID":"288c920e-ace3-4b8a-b5fd-2c94dfb4f056","Type":"ContainerStarted","Data":"c80482b6bc75bd9cb4f92a528dfa77b0020ec854de34b598c7b9adb67f4c566f"}
Oct 30 13:52:55.641895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:55.641854    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6" event={"ID":"288c920e-ace3-4b8a-b5fd-2c94dfb4f056","Type":"ContainerStarted","Data":"1ba1ff56beb6dd80909dd59a2af3d6e7dbf110cd35c4a854cda8859b7f446a63"}
Oct 30 13:52:55.755239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:55.751197    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79100f5f-7848-488a-819a-eb9227e0e055" path="/var/lib/kubelet/pods/79100f5f-7848-488a-819a-eb9227e0e055/volumes"
Oct 30 13:52:56.651442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:56.651398    4993 generic.go:356] "Generic (PLEG): container finished" podID="288c920e-ace3-4b8a-b5fd-2c94dfb4f056" containerID="c80482b6bc75bd9cb4f92a528dfa77b0020ec854de34b598c7b9adb67f4c566f" exitCode=0
Oct 30 13:52:56.651989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:56.651490    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6" event={"ID":"288c920e-ace3-4b8a-b5fd-2c94dfb4f056","Type":"ContainerDied","Data":"c80482b6bc75bd9cb4f92a528dfa77b0020ec854de34b598c7b9adb67f4c566f"}
Oct 30 13:52:57.851757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:57.851722    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:57.907362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:57.907324    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-kube-api-access-q822r\" (UniqueName: \"kubernetes.io/projected/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-kube-api-access-q822r\") pod \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") "
Oct 30 13:52:57.907584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:57.907540    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-openshift-ca\") pod \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") "
Oct 30 13:52:57.907645 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:57.907607    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-xks-ca\") pod \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\" (UID: \"288c920e-ace3-4b8a-b5fd-2c94dfb4f056\") "
Oct 30 13:52:57.907968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:57.907936    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-openshift-ca" pod "288c920e-ace3-4b8a-b5fd-2c94dfb4f056" (UID: "288c920e-ace3-4b8a-b5fd-2c94dfb4f056"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:57.908254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:57.908191    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:57.915361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:57.915325    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-xks-ca" pod "288c920e-ace3-4b8a-b5fd-2c94dfb4f056" (UID: "288c920e-ace3-4b8a-b5fd-2c94dfb4f056"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:57.917518 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:57.917444    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-kube-api-access-q822r" pod "288c920e-ace3-4b8a-b5fd-2c94dfb4f056" (UID: "288c920e-ace3-4b8a-b5fd-2c94dfb4f056"). InnerVolumeSpecName "kube-api-access-q822r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:58.009617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:58.009571    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q822r\" (UniqueName: \"kubernetes.io/projected/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-kube-api-access-q822r\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:58.009617 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:58.009609    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288c920e-ace3-4b8a-b5fd-2c94dfb4f056-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:52:58.675289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:58.675243    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6" event={"ID":"288c920e-ace3-4b8a-b5fd-2c94dfb4f056","Type":"ContainerDied","Data":"1ba1ff56beb6dd80909dd59a2af3d6e7dbf110cd35c4a854cda8859b7f446a63"}
Oct 30 13:52:58.675490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:58.675269    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"
Oct 30 13:52:58.675490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:52:58.675342    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ba1ff56beb6dd80909dd59a2af3d6e7dbf110cd35c4a854cda8859b7f446a63"
Oct 30 13:53:03.827769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:03.827722    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq" podUID="7b3ed3ca-445a-4a0b-ae08-a136c26c55de" containerName="main" containerID="cri-o://0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28" gracePeriod=45
Oct 30 13:53:04.338909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.332377    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:53:04.588075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.587966    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-dshm\") pod \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") "
Oct 30 13:53:04.588075 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.588043    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tmp-dir\") pod \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") "
Oct 30 13:53:04.588535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.588099    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-kube-api-access-5s88v\" (UniqueName: \"kubernetes.io/projected/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-kube-api-access-5s88v\") pod \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") "
Oct 30 13:53:04.588535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.588134    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-model-cache\") pod \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") "
Oct 30 13:53:04.588535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.588169    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tls-certs\") pod \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") "
Oct 30 13:53:04.588535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.588373    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-home\") pod \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\" (UID: \"7b3ed3ca-445a-4a0b-ae08-a136c26c55de\") "
Oct 30 13:53:04.590176 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.590137    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-model-cache" pod "7b3ed3ca-445a-4a0b-ae08-a136c26c55de" (UID: "7b3ed3ca-445a-4a0b-ae08-a136c26c55de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.590628 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.590599    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-home" pod "7b3ed3ca-445a-4a0b-ae08-a136c26c55de" (UID: "7b3ed3ca-445a-4a0b-ae08-a136c26c55de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.591149 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.591122    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tmp-dir" pod "7b3ed3ca-445a-4a0b-ae08-a136c26c55de" (UID: "7b3ed3ca-445a-4a0b-ae08-a136c26c55de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.592068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.592043    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-kube-api-access-5s88v" pod "7b3ed3ca-445a-4a0b-ae08-a136c26c55de" (UID: "7b3ed3ca-445a-4a0b-ae08-a136c26c55de"). InnerVolumeSpecName "kube-api-access-5s88v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:04.594150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.594081    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-dshm" pod "7b3ed3ca-445a-4a0b-ae08-a136c26c55de" (UID: "7b3ed3ca-445a-4a0b-ae08-a136c26c55de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.594795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.594772    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tls-certs" pod "7b3ed3ca-445a-4a0b-ae08-a136c26c55de" (UID: "7b3ed3ca-445a-4a0b-ae08-a136c26c55de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:04.690964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.690911    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.690964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.690952    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.690964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.690966    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s88v\" (UniqueName: \"kubernetes.io/projected/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-kube-api-access-5s88v\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.691355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.690981    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.691355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.690995    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.691355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.691006    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ed3ca-445a-4a0b-ae08-a136c26c55de-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.730664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.730605    4993 generic.go:356] "Generic (PLEG): container finished" podID="7b3ed3ca-445a-4a0b-ae08-a136c26c55de" containerID="0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28" exitCode=0
Oct 30 13:53:04.730885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.730673    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"
Oct 30 13:53:04.730885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.730693    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq" event={"ID":"7b3ed3ca-445a-4a0b-ae08-a136c26c55de","Type":"ContainerDied","Data":"0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28"}
Oct 30 13:53:04.730885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.730731    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq" event={"ID":"7b3ed3ca-445a-4a0b-ae08-a136c26c55de","Type":"ContainerDied","Data":"6d210e9f9bfc8333ff857fbc165da0ebb70bd4ef3064944a0d9ec30602611162"}
Oct 30 13:53:04.730885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.730752    4993 scope.go:122] "RemoveContainer" containerID="0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28"
Oct 30 13:53:04.745413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.745377    4993 scope.go:122] "RemoveContainer" containerID="0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28"
Oct 30 13:53:04.745865 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:53:04.745834    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28\": container with ID starting with 0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28 not found: ID does not exist" containerID="0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28"
Oct 30 13:53:04.745978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.745880    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28"} err="failed to get container status \"0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28\": rpc error: code = NotFound desc = could not find container \"0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28\": container with ID starting with 0ee7152f3462287217652db0f6fd907dae73c429a87f6fb8c52b8c015aa36b28 not found: ID does not exist"
Oct 30 13:53:04.797988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.797574    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"]
Oct 30 13:53:04.803439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:04.803396    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/sub-test-model-7a8b38-kserve-56954f5cf5-k8lvq"]
Oct 30 13:53:05.692880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:05.692843    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3ed3ca-445a-4a0b-ae08-a136c26c55de" path="/var/lib/kubelet/pods/7b3ed3ca-445a-4a0b-ae08-a136c26c55de/volumes"
Oct 30 13:53:08.574977 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:53:08.574913    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:53:09.029728 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.029684    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz" podUID="deb93120-9646-4149-8c6d-01f6b52a2c37" containerName="main" containerID="cri-o://6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f" gracePeriod=45
Oct 30 13:53:09.337264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.337192    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:53:09.389009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.388963    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-home\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-home\") pod \"deb93120-9646-4149-8c6d-01f6b52a2c37\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") "
Oct 30 13:53:09.389279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.389191    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-dshm\") pod \"deb93120-9646-4149-8c6d-01f6b52a2c37\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") "
Oct 30 13:53:09.389279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.389261    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-model-cache\") pod \"deb93120-9646-4149-8c6d-01f6b52a2c37\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") "
Oct 30 13:53:09.389453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.389334    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb93120-9646-4149-8c6d-01f6b52a2c37-kube-api-access-zxt7b\" (UniqueName: \"kubernetes.io/projected/deb93120-9646-4149-8c6d-01f6b52a2c37-kube-api-access-zxt7b\") pod \"deb93120-9646-4149-8c6d-01f6b52a2c37\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") "
Oct 30 13:53:09.389453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.389373    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb93120-9646-4149-8c6d-01f6b52a2c37-tls-certs\" (UniqueName: \"kubernetes.io/secret/deb93120-9646-4149-8c6d-01f6b52a2c37-tls-certs\") pod \"deb93120-9646-4149-8c6d-01f6b52a2c37\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") "
Oct 30 13:53:09.389453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.389418    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-tmp-dir\") pod \"deb93120-9646-4149-8c6d-01f6b52a2c37\" (UID: \"deb93120-9646-4149-8c6d-01f6b52a2c37\") "
Oct 30 13:53:09.393104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.393068    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-tmp-dir" pod "deb93120-9646-4149-8c6d-01f6b52a2c37" (UID: "deb93120-9646-4149-8c6d-01f6b52a2c37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:09.393306 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.393285    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-home" pod "deb93120-9646-4149-8c6d-01f6b52a2c37" (UID: "deb93120-9646-4149-8c6d-01f6b52a2c37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:09.394337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.394308    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.394337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.394337    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.396395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.396363    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-dshm" pod "deb93120-9646-4149-8c6d-01f6b52a2c37" (UID: "deb93120-9646-4149-8c6d-01f6b52a2c37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:09.397177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.397154    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb93120-9646-4149-8c6d-01f6b52a2c37-kube-api-access-zxt7b" pod "deb93120-9646-4149-8c6d-01f6b52a2c37" (UID: "deb93120-9646-4149-8c6d-01f6b52a2c37"). InnerVolumeSpecName "kube-api-access-zxt7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:09.397637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.397615    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-model-cache" pod "deb93120-9646-4149-8c6d-01f6b52a2c37" (UID: "deb93120-9646-4149-8c6d-01f6b52a2c37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:09.398761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.398732    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb93120-9646-4149-8c6d-01f6b52a2c37-tls-certs" pod "deb93120-9646-4149-8c6d-01f6b52a2c37" (UID: "deb93120-9646-4149-8c6d-01f6b52a2c37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:09.495789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.495736    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.495789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.495777    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb93120-9646-4149-8c6d-01f6b52a2c37-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.495789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.495789    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxt7b\" (UniqueName: \"kubernetes.io/projected/deb93120-9646-4149-8c6d-01f6b52a2c37-kube-api-access-zxt7b\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.495789 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.495799    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb93120-9646-4149-8c6d-01f6b52a2c37-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.791372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.791329    4993 generic.go:356] "Generic (PLEG): container finished" podID="deb93120-9646-4149-8c6d-01f6b52a2c37" containerID="6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f" exitCode=0
Oct 30 13:53:09.791866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.791376    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"
Oct 30 13:53:09.791866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.791405    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz" event={"ID":"deb93120-9646-4149-8c6d-01f6b52a2c37","Type":"ContainerDied","Data":"6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f"}
Oct 30 13:53:09.791866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.791468    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz" event={"ID":"deb93120-9646-4149-8c6d-01f6b52a2c37","Type":"ContainerDied","Data":"824a40d9d4acd0de8558b3ed369788999758711f00265b6c5c9b028e0c2e5c95"}
Oct 30 13:53:09.791866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.791489    4993 scope.go:122] "RemoveContainer" containerID="6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f"
Oct 30 13:53:09.802643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.802614    4993 scope.go:122] "RemoveContainer" containerID="6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f"
Oct 30 13:53:09.803025 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:53:09.802996    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f\": container with ID starting with 6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f not found: ID does not exist" containerID="6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f"
Oct 30 13:53:09.803129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.803040    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f"} err="failed to get container status \"6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f\": rpc error: code = NotFound desc = could not find container \"6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f\": container with ID starting with 6c80d16726ee90cc0d777be6ce8eded71f84600579cff46958ca5f93ca62567f not found: ID does not exist"
Oct 30 13:53:09.847652 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.847616    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"]
Oct 30 13:53:09.858443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:09.858408    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/sub-test-model-98dabb-kserve-58cd4f75cc-g6wpz"]
Oct 30 13:53:11.693251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:11.693200    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb93120-9646-4149-8c6d-01f6b52a2c37" path="/var/lib/kubelet/pods/deb93120-9646-4149-8c6d-01f6b52a2c37/volumes"
Oct 30 13:53:13.544061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544017    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"]
Oct 30 13:53:13.544596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544580    4993 cpu_manager.go:414] "containerMap: removing container" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy"
Oct 30 13:53:13.544661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544597    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy"
Oct 30 13:53:13.544661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544628    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7b3ed3ca-445a-4a0b-ae08-a136c26c55de" containerName="main"
Oct 30 13:53:13.544661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544635    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3ed3ca-445a-4a0b-ae08-a136c26c55de" containerName="main"
Oct 30 13:53:13.544661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544651    4993 cpu_manager.go:414] "containerMap: removing container" podUID="288c920e-ace3-4b8a-b5fd-2c94dfb4f056" containerName="revoke-keys"
Oct 30 13:53:13.544661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544656    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288c920e-ace3-4b8a-b5fd-2c94dfb4f056" containerName="revoke-keys"
Oct 30 13:53:13.544833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544672    4993 cpu_manager.go:414] "containerMap: removing container" podUID="deb93120-9646-4149-8c6d-01f6b52a2c37" containerName="main"
Oct 30 13:53:13.544833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544677    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb93120-9646-4149-8c6d-01f6b52a2c37" containerName="main"
Oct 30 13:53:13.544833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544767    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3ed3ca-445a-4a0b-ae08-a136c26c55de" containerName="main"
Oct 30 13:53:13.544833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544776    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="79100f5f-7848-488a-819a-eb9227e0e055" containerName="istio-proxy"
Oct 30 13:53:13.544833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544784    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="288c920e-ace3-4b8a-b5fd-2c94dfb4f056" containerName="revoke-keys"
Oct 30 13:53:13.544833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.544794    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb93120-9646-4149-8c6d-01f6b52a2c37" containerName="main"
Oct 30 13:53:13.549745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.549718    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.576491 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.576450    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:53:13.576689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.576548    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:53:13.583913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.583867    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"]
Oct 30 13:53:13.605888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.605852    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:13.606083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.605986    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:53:13.673104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.673059    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.673312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.673201    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpb2h\" (UniqueName: \"kubernetes.io/projected/6aaf2d54-a366-432d-ad2e-0475a16412cb-kube-api-access-qpb2h\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.673427 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.673410    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.775002 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.774957    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.775333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.775079    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpb2h\" (UniqueName: \"kubernetes.io/projected/6aaf2d54-a366-432d-ad2e-0475a16412cb-kube-api-access-qpb2h\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.775333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.775264    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.777182 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.777145    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.800480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.800080    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.802975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.802925    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpb2h\" (UniqueName: \"kubernetes.io/projected/6aaf2d54-a366-432d-ad2e-0475a16412cb-kube-api-access-qpb2h\") pod \"maas-api-revoke-sub-60c8aa88-5tc8l\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:13.865382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:13.865340    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:14.174135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:14.174101    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"]
Oct 30 13:53:14.174947 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:53:14.174900    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaf2d54_a366_432d_ad2e_0475a16412cb.slice/crio-8d9ca698defe69b0826ca33ba5830c00ffb9f73ca41acdb36afec398eb51e8e6 WatchSource:0}: Error finding container 8d9ca698defe69b0826ca33ba5830c00ffb9f73ca41acdb36afec398eb51e8e6: Status 404 returned error can't find the container with id 8d9ca698defe69b0826ca33ba5830c00ffb9f73ca41acdb36afec398eb51e8e6
Oct 30 13:53:14.832780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:14.832739    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l" event={"ID":"6aaf2d54-a366-432d-ad2e-0475a16412cb","Type":"ContainerStarted","Data":"6744ef55e598ceccb8242dd1e2248a46198ca27955b0547f761f9cb90380a829"}
Oct 30 13:53:14.833196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:14.832797    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l" event={"ID":"6aaf2d54-a366-432d-ad2e-0475a16412cb","Type":"ContainerStarted","Data":"8d9ca698defe69b0826ca33ba5830c00ffb9f73ca41acdb36afec398eb51e8e6"}
Oct 30 13:53:14.873249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:14.873125    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l" podStartSLOduration=1.873106516 podStartE2EDuration="1.873106516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:14.871602229 +0000 UTC m=+4859.347321838" watchObservedRunningTime="2026-10-30 13:53:14.873106516 +0000 UTC m=+4859.348826128"
Oct 30 13:53:15.842989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:15.842954    4993 generic.go:356] "Generic (PLEG): container finished" podID="6aaf2d54-a366-432d-ad2e-0475a16412cb" containerID="6744ef55e598ceccb8242dd1e2248a46198ca27955b0547f761f9cb90380a829" exitCode=0
Oct 30 13:53:15.843557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:15.843034    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l" event={"ID":"6aaf2d54-a366-432d-ad2e-0475a16412cb","Type":"ContainerDied","Data":"6744ef55e598ceccb8242dd1e2248a46198ca27955b0547f761f9cb90380a829"}
Oct 30 13:53:17.023398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.023363    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:17.125653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.125542    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-xks-ca\") pod \"6aaf2d54-a366-432d-ad2e-0475a16412cb\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") "
Oct 30 13:53:17.125653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.125611    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-openshift-ca\") pod \"6aaf2d54-a366-432d-ad2e-0475a16412cb\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") "
Oct 30 13:53:17.125903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.125657    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aaf2d54-a366-432d-ad2e-0475a16412cb-kube-api-access-qpb2h\" (UniqueName: \"kubernetes.io/projected/6aaf2d54-a366-432d-ad2e-0475a16412cb-kube-api-access-qpb2h\") pod \"6aaf2d54-a366-432d-ad2e-0475a16412cb\" (UID: \"6aaf2d54-a366-432d-ad2e-0475a16412cb\") "
Oct 30 13:53:17.126010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.125980    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-openshift-ca" pod "6aaf2d54-a366-432d-ad2e-0475a16412cb" (UID: "6aaf2d54-a366-432d-ad2e-0475a16412cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:17.126361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.126337    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:17.128052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.128022    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaf2d54-a366-432d-ad2e-0475a16412cb-kube-api-access-qpb2h" pod "6aaf2d54-a366-432d-ad2e-0475a16412cb" (UID: "6aaf2d54-a366-432d-ad2e-0475a16412cb"). InnerVolumeSpecName "kube-api-access-qpb2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:17.128279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.128255    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-xks-ca" pod "6aaf2d54-a366-432d-ad2e-0475a16412cb" (UID: "6aaf2d54-a366-432d-ad2e-0475a16412cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:17.228046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.227994    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2d54-a366-432d-ad2e-0475a16412cb-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:17.228046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.228036    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpb2h\" (UniqueName: \"kubernetes.io/projected/6aaf2d54-a366-432d-ad2e-0475a16412cb-kube-api-access-qpb2h\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:17.862376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.862338    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"
Oct 30 13:53:17.862613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.862355    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l" event={"ID":"6aaf2d54-a366-432d-ad2e-0475a16412cb","Type":"ContainerDied","Data":"8d9ca698defe69b0826ca33ba5830c00ffb9f73ca41acdb36afec398eb51e8e6"}
Oct 30 13:53:17.862613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:17.862437    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d9ca698defe69b0826ca33ba5830c00ffb9f73ca41acdb36afec398eb51e8e6"
Oct 30 13:53:19.215054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.215013    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"]
Oct 30 13:53:19.215676 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.215657    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6aaf2d54-a366-432d-ad2e-0475a16412cb" containerName="revoke-keys"
Oct 30 13:53:19.215719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.215677    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aaf2d54-a366-432d-ad2e-0475a16412cb" containerName="revoke-keys"
Oct 30 13:53:19.215797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.215788    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aaf2d54-a366-432d-ad2e-0475a16412cb" containerName="revoke-keys"
Oct 30 13:53:19.219372 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.219348    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.235631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.235596    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"]
Oct 30 13:53:19.239848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.239815    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.243503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.243471    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:53:19.249777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.249748    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:19.250131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.250054    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:53:19.250131 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.250058    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:53:19.256975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.256931    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"]
Oct 30 13:53:19.283244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.283165    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"]
Oct 30 13:53:19.359291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.359247    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.359291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.359305    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6z4\" (UniqueName: \"kubernetes.io/projected/4d456c31-7321-471e-bf1d-90abe178e46e-kube-api-access-7v6z4\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.359611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.359438    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.359611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.359508    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.359611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.359561    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.359787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.359638    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5g6\" (UniqueName: \"kubernetes.io/projected/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-kube-api-access-cg5g6\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.461481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.461428    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.461481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.461486    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6z4\" (UniqueName: \"kubernetes.io/projected/4d456c31-7321-471e-bf1d-90abe178e46e-kube-api-access-7v6z4\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.461774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.461535    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.461774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.461577    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.461774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.461598    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.461774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.461664    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5g6\" (UniqueName: \"kubernetes.io/projected/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-kube-api-access-cg5g6\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.462430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.462394    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.462730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.462707    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.464338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.464304    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.464587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.464564    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.486733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.486693    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6z4\" (UniqueName: \"kubernetes.io/projected/4d456c31-7321-471e-bf1d-90abe178e46e-kube-api-access-7v6z4\") pod \"maas-api-revoke-sub-ef524820-6z89w\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.486733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.486724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5g6\" (UniqueName: \"kubernetes.io/projected/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-kube-api-access-cg5g6\") pod \"maas-api-revoke-sub-15d99453-2j745\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.535544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.535504    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:19.581037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.580991    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:19.850084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.850046    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"]
Oct 30 13:53:19.850951 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:53:19.850923    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440c9ea0_5b0f_479f_9636_fe573c3f2dfc.slice/crio-5b23f6c146d6da8c5743ec90861760ccb542f22f38da8da2f4ba56bbf771d72e WatchSource:0}: Error finding container 5b23f6c146d6da8c5743ec90861760ccb542f22f38da8da2f4ba56bbf771d72e: Status 404 returned error can't find the container with id 5b23f6c146d6da8c5743ec90861760ccb542f22f38da8da2f4ba56bbf771d72e
Oct 30 13:53:19.884432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.884388    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745" event={"ID":"440c9ea0-5b0f-479f-9636-fe573c3f2dfc","Type":"ContainerStarted","Data":"5b23f6c146d6da8c5743ec90861760ccb542f22f38da8da2f4ba56bbf771d72e"}
Oct 30 13:53:19.905293 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:19.905257    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"]
Oct 30 13:53:19.906964 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:53:19.906931    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d456c31_7321_471e_bf1d_90abe178e46e.slice/crio-e53beb51b597d3aa0e634092a651c2a6e64ab9776bbc11ee44f13a06b916c9bc WatchSource:0}: Error finding container e53beb51b597d3aa0e634092a651c2a6e64ab9776bbc11ee44f13a06b916c9bc: Status 404 returned error can't find the container with id e53beb51b597d3aa0e634092a651c2a6e64ab9776bbc11ee44f13a06b916c9bc
Oct 30 13:53:20.896073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:20.896025    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745" event={"ID":"440c9ea0-5b0f-479f-9636-fe573c3f2dfc","Type":"ContainerStarted","Data":"ad6cbdf1cb20fa8b7721fa054bdcbeddfed57703723462fc5ebcbc0485f72fd1"}
Oct 30 13:53:20.898010 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:20.897981    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w" event={"ID":"4d456c31-7321-471e-bf1d-90abe178e46e","Type":"ContainerStarted","Data":"5a601819cd444142ca091143c657eee9fc9b4c236c0e87d10949a85154035ad5"}
Oct 30 13:53:20.898165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:20.898031    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w" event={"ID":"4d456c31-7321-471e-bf1d-90abe178e46e","Type":"ContainerStarted","Data":"e53beb51b597d3aa0e634092a651c2a6e64ab9776bbc11ee44f13a06b916c9bc"}
Oct 30 13:53:20.938793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:20.938730    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745" podStartSLOduration=1.938716485 podStartE2EDuration="1.938716485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:20.937475891 +0000 UTC m=+4865.413195501" watchObservedRunningTime="2026-10-30 13:53:20.938716485 +0000 UTC m=+4865.414436094"
Oct 30 13:53:20.980564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:20.980505    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w" podStartSLOduration=1.9804878750000001 podStartE2EDuration="1.980487875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:20.9770848 +0000 UTC m=+4865.452804424" watchObservedRunningTime="2026-10-30 13:53:20.980487875 +0000 UTC m=+4865.456207488"
Oct 30 13:53:21.906778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:21.906741    4993 generic.go:356] "Generic (PLEG): container finished" podID="440c9ea0-5b0f-479f-9636-fe573c3f2dfc" containerID="ad6cbdf1cb20fa8b7721fa054bdcbeddfed57703723462fc5ebcbc0485f72fd1" exitCode=0
Oct 30 13:53:21.907256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:21.906824    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745" event={"ID":"440c9ea0-5b0f-479f-9636-fe573c3f2dfc","Type":"ContainerDied","Data":"ad6cbdf1cb20fa8b7721fa054bdcbeddfed57703723462fc5ebcbc0485f72fd1"}
Oct 30 13:53:21.908627 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:21.908604    4993 generic.go:356] "Generic (PLEG): container finished" podID="4d456c31-7321-471e-bf1d-90abe178e46e" containerID="5a601819cd444142ca091143c657eee9fc9b4c236c0e87d10949a85154035ad5" exitCode=0
Oct 30 13:53:21.908761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:21.908686    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w" event={"ID":"4d456c31-7321-471e-bf1d-90abe178e46e","Type":"ContainerDied","Data":"5a601819cd444142ca091143c657eee9fc9b4c236c0e87d10949a85154035ad5"}
Oct 30 13:53:23.214325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.214289    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:23.218346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.218317    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:23.224716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.224682    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d456c31-7321-471e-bf1d-90abe178e46e-kube-api-access-7v6z4\" (UniqueName: \"kubernetes.io/projected/4d456c31-7321-471e-bf1d-90abe178e46e-kube-api-access-7v6z4\") pod \"4d456c31-7321-471e-bf1d-90abe178e46e\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") "
Oct 30 13:53:23.224888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.224820    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-openshift-ca\") pod \"4d456c31-7321-471e-bf1d-90abe178e46e\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") "
Oct 30 13:53:23.225034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.225014    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-xks-ca\") pod \"4d456c31-7321-471e-bf1d-90abe178e46e\" (UID: \"4d456c31-7321-471e-bf1d-90abe178e46e\") "
Oct 30 13:53:23.225266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.225241    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-openshift-ca" pod "4d456c31-7321-471e-bf1d-90abe178e46e" (UID: "4d456c31-7321-471e-bf1d-90abe178e46e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:23.225541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.225524    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:23.226944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.226908    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d456c31-7321-471e-bf1d-90abe178e46e-kube-api-access-7v6z4" pod "4d456c31-7321-471e-bf1d-90abe178e46e" (UID: "4d456c31-7321-471e-bf1d-90abe178e46e"). InnerVolumeSpecName "kube-api-access-7v6z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:23.227051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.226997    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-xks-ca" pod "4d456c31-7321-471e-bf1d-90abe178e46e" (UID: "4d456c31-7321-471e-bf1d-90abe178e46e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:23.326598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.326556    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-openshift-ca\") pod \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") "
Oct 30 13:53:23.326813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.326718    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-xks-ca\") pod \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") "
Oct 30 13:53:23.326813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.326799    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-kube-api-access-cg5g6\" (UniqueName: \"kubernetes.io/projected/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-kube-api-access-cg5g6\") pod \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\" (UID: \"440c9ea0-5b0f-479f-9636-fe573c3f2dfc\") "
Oct 30 13:53:23.327070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.327037    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-openshift-ca" pod "440c9ea0-5b0f-479f-9636-fe573c3f2dfc" (UID: "440c9ea0-5b0f-479f-9636-fe573c3f2dfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:23.327381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.327360    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d456c31-7321-471e-bf1d-90abe178e46e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:23.327381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.327380    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6z4\" (UniqueName: \"kubernetes.io/projected/4d456c31-7321-471e-bf1d-90abe178e46e-kube-api-access-7v6z4\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:23.327547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.327390    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:23.329027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.329000    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-xks-ca" pod "440c9ea0-5b0f-479f-9636-fe573c3f2dfc" (UID: "440c9ea0-5b0f-479f-9636-fe573c3f2dfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:23.329027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.329011    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-kube-api-access-cg5g6" pod "440c9ea0-5b0f-479f-9636-fe573c3f2dfc" (UID: "440c9ea0-5b0f-479f-9636-fe573c3f2dfc"). InnerVolumeSpecName "kube-api-access-cg5g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:23.428531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.428434    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:23.428531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.428467    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg5g6\" (UniqueName: \"kubernetes.io/projected/440c9ea0-5b0f-479f-9636-fe573c3f2dfc-kube-api-access-cg5g6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:23.925559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.925520    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"
Oct 30 13:53:23.925729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.925554    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745" event={"ID":"440c9ea0-5b0f-479f-9636-fe573c3f2dfc","Type":"ContainerDied","Data":"5b23f6c146d6da8c5743ec90861760ccb542f22f38da8da2f4ba56bbf771d72e"}
Oct 30 13:53:23.925729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.925611    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b23f6c146d6da8c5743ec90861760ccb542f22f38da8da2f4ba56bbf771d72e"
Oct 30 13:53:23.927047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.927023    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"
Oct 30 13:53:23.927217 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.927069    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w" event={"ID":"4d456c31-7321-471e-bf1d-90abe178e46e","Type":"ContainerDied","Data":"e53beb51b597d3aa0e634092a651c2a6e64ab9776bbc11ee44f13a06b916c9bc"}
Oct 30 13:53:23.927217 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.927115    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e53beb51b597d3aa0e634092a651c2a6e64ab9776bbc11ee44f13a06b916c9bc"
Oct 30 13:53:23.972490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.972382    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"]
Oct 30 13:53:23.972959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.972944    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4d456c31-7321-471e-bf1d-90abe178e46e" containerName="revoke-keys"
Oct 30 13:53:23.973029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.972960    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d456c31-7321-471e-bf1d-90abe178e46e" containerName="revoke-keys"
Oct 30 13:53:23.973029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.973014    4993 cpu_manager.go:414] "containerMap: removing container" podUID="440c9ea0-5b0f-479f-9636-fe573c3f2dfc" containerName="revoke-keys"
Oct 30 13:53:23.973029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.973021    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440c9ea0-5b0f-479f-9636-fe573c3f2dfc" containerName="revoke-keys"
Oct 30 13:53:23.973220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.973178    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d456c31-7321-471e-bf1d-90abe178e46e" containerName="revoke-keys"
Oct 30 13:53:23.973220 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.973218    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="440c9ea0-5b0f-479f-9636-fe573c3f2dfc" containerName="revoke-keys"
Oct 30 13:53:23.977417 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.977386    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:23.993542 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.993512    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:23.994756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.994725    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:53:23.994893 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:23.994778    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:53:24.005347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.005311    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"]
Oct 30 13:53:24.037067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.037022    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.037351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.037320    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.037460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.037432    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n49t\" (UniqueName: \"kubernetes.io/projected/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-kube-api-access-9n49t\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.091646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.091610    4993 scope.go:122] "RemoveContainer" containerID="5129916921da61b51274e2c627a5850373afb5c97a29708f9c79aee96d76630b"
Oct 30 13:53:24.093625 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.093601    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:53:24.139088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.139051    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.139270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.139129    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n49t\" (UniqueName: \"kubernetes.io/projected/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-kube-api-access-9n49t\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.139659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.139631    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.139891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.139874    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.142129 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.142108    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.162855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.162813    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n49t\" (UniqueName: \"kubernetes.io/projected/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-kube-api-access-9n49t\") pod \"maas-api-revoke-sub-462560f3-ts2q5\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.294146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.294101    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:24.606804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.606761    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"]
Oct 30 13:53:24.609403 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:53:24.609369    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b662f71_a8fc_4b3c_bcdd_fbb7f4ea9807.slice/crio-29350b0ddc303be46f51a56fa3c4f6e855bc6e9b740b51d53656726c5a324d66 WatchSource:0}: Error finding container 29350b0ddc303be46f51a56fa3c4f6e855bc6e9b740b51d53656726c5a324d66: Status 404 returned error can't find the container with id 29350b0ddc303be46f51a56fa3c4f6e855bc6e9b740b51d53656726c5a324d66
Oct 30 13:53:24.936368 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:24.936271    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5" event={"ID":"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807","Type":"ContainerStarted","Data":"29350b0ddc303be46f51a56fa3c4f6e855bc6e9b740b51d53656726c5a324d66"}
Oct 30 13:53:25.946316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:25.946277    4993 generic.go:356] "Generic (PLEG): container finished" podID="0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807" containerID="6b8051de9d50afcd782156a57de861cae44fa885198d623d9ffaf13bde590d42" exitCode=0
Oct 30 13:53:25.946729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:25.946389    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5" event={"ID":"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807","Type":"ContainerDied","Data":"6b8051de9d50afcd782156a57de861cae44fa885198d623d9ffaf13bde590d42"}
Oct 30 13:53:27.125620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.125590    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:27.182475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.182420    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-openshift-ca\") pod \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") "
Oct 30 13:53:27.182688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.182503    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-xks-ca\") pod \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") "
Oct 30 13:53:27.182751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.182727    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-kube-api-access-9n49t\" (UniqueName: \"kubernetes.io/projected/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-kube-api-access-9n49t\") pod \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\" (UID: \"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807\") "
Oct 30 13:53:27.182805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.182779    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-openshift-ca" pod "0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807" (UID: "0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:27.183282 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.183259    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:27.184764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.184737    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-xks-ca" pod "0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807" (UID: "0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:27.185171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.185155    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-kube-api-access-9n49t" pod "0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807" (UID: "0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807"). InnerVolumeSpecName "kube-api-access-9n49t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:27.284694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.284648    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:27.284694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:27.284683    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n49t\" (UniqueName: \"kubernetes.io/projected/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807-kube-api-access-9n49t\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:28.019228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:28.019112    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5" event={"ID":"0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807","Type":"ContainerDied","Data":"29350b0ddc303be46f51a56fa3c4f6e855bc6e9b740b51d53656726c5a324d66"}
Oct 30 13:53:28.019228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:28.019184    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29350b0ddc303be46f51a56fa3c4f6e855bc6e9b740b51d53656726c5a324d66"
Oct 30 13:53:28.019512 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:28.019277    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"
Oct 30 13:53:34.861351 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.861305    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"]
Oct 30 13:53:34.862120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.862099    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807" containerName="revoke-keys"
Oct 30 13:53:34.862186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.862122    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807" containerName="revoke-keys"
Oct 30 13:53:34.862367 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.862352    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807" containerName="revoke-keys"
Oct 30 13:53:34.866770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.866746    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:34.893744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.893710    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:34.893940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.893762    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:53:34.893940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.893819    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:53:34.893940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.893880    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:53:34.899067 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.899027    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"]
Oct 30 13:53:34.939560 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.939513    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:34.939779 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.939753    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:34.939857 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:34.939802    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9vmw\" (UniqueName: \"kubernetes.io/projected/748ca8b1-9602-4a02-a162-d222afb56e5e-kube-api-access-m9vmw\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:35.041591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:35.041549    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:35.041790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:35.041752    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:35.041851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:35.041797    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9vmw\" (UniqueName: \"kubernetes.io/projected/748ca8b1-9602-4a02-a162-d222afb56e5e-kube-api-access-m9vmw\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:35.042442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:35.042415    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:35.044343 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:35.044320    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:35.098256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:35.097336    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9vmw\" (UniqueName: \"kubernetes.io/projected/748ca8b1-9602-4a02-a162-d222afb56e5e-kube-api-access-m9vmw\") pod \"maas-api-revoke-sub-0c9ad015-6l9pq\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:35.182735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:35.182643    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:35.496253 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:53:35.495641    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748ca8b1_9602_4a02_a162_d222afb56e5e.slice/crio-55bdd0e15fecc985e5cb19c0985f9ae03c60c0b2b7a6cf82063ceabe1ee55d85 WatchSource:0}: Error finding container 55bdd0e15fecc985e5cb19c0985f9ae03c60c0b2b7a6cf82063ceabe1ee55d85: Status 404 returned error can't find the container with id 55bdd0e15fecc985e5cb19c0985f9ae03c60c0b2b7a6cf82063ceabe1ee55d85
Oct 30 13:53:35.496253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:35.495656    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"]
Oct 30 13:53:36.128050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:36.127942    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq" event={"ID":"748ca8b1-9602-4a02-a162-d222afb56e5e","Type":"ContainerStarted","Data":"c9f2e3ce433f1e1de6fca4994bd7640b1fe0397b82dc1f4717cce7364316f521"}
Oct 30 13:53:36.128050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:36.128015    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq" event={"ID":"748ca8b1-9602-4a02-a162-d222afb56e5e","Type":"ContainerStarted","Data":"55bdd0e15fecc985e5cb19c0985f9ae03c60c0b2b7a6cf82063ceabe1ee55d85"}
Oct 30 13:53:36.171294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:36.171235    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq" podStartSLOduration=2.17121844 podStartE2EDuration="2.17121844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:36.167330059 +0000 UTC m=+4880.643049671" watchObservedRunningTime="2026-10-30 13:53:36.17121844 +0000 UTC m=+4880.646938041"
Oct 30 13:53:37.121575 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:37.121115    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"]
Oct 30 13:53:37.134259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:37.134221    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-acf9e2-4jdx6"]
Oct 30 13:53:37.148430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:37.148393    4993 generic.go:356] "Generic (PLEG): container finished" podID="748ca8b1-9602-4a02-a162-d222afb56e5e" containerID="c9f2e3ce433f1e1de6fca4994bd7640b1fe0397b82dc1f4717cce7364316f521" exitCode=0
Oct 30 13:53:37.148601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:37.148464    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq" event={"ID":"748ca8b1-9602-4a02-a162-d222afb56e5e","Type":"ContainerDied","Data":"c9f2e3ce433f1e1de6fca4994bd7640b1fe0397b82dc1f4717cce7364316f521"}
Oct 30 13:53:37.694761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:37.694721    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288c920e-ace3-4b8a-b5fd-2c94dfb4f056" path="/var/lib/kubelet/pods/288c920e-ace3-4b8a-b5fd-2c94dfb4f056/volumes"
Oct 30 13:53:38.330973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.330942    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:38.430863 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.430766    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-openshift-ca\") pod \"748ca8b1-9602-4a02-a162-d222afb56e5e\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") "
Oct 30 13:53:38.431028 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.431002    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748ca8b1-9602-4a02-a162-d222afb56e5e-kube-api-access-m9vmw\" (UniqueName: \"kubernetes.io/projected/748ca8b1-9602-4a02-a162-d222afb56e5e-kube-api-access-m9vmw\") pod \"748ca8b1-9602-4a02-a162-d222afb56e5e\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") "
Oct 30 13:53:38.431113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.431076    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-xks-ca\") pod \"748ca8b1-9602-4a02-a162-d222afb56e5e\" (UID: \"748ca8b1-9602-4a02-a162-d222afb56e5e\") "
Oct 30 13:53:38.431706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.431277    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-openshift-ca" pod "748ca8b1-9602-4a02-a162-d222afb56e5e" (UID: "748ca8b1-9602-4a02-a162-d222afb56e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:38.431706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.431670    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:38.433686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.433661    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-xks-ca" pod "748ca8b1-9602-4a02-a162-d222afb56e5e" (UID: "748ca8b1-9602-4a02-a162-d222afb56e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:38.439054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.439028    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748ca8b1-9602-4a02-a162-d222afb56e5e-kube-api-access-m9vmw" pod "748ca8b1-9602-4a02-a162-d222afb56e5e" (UID: "748ca8b1-9602-4a02-a162-d222afb56e5e"). InnerVolumeSpecName "kube-api-access-m9vmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:38.532638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.532583    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9vmw\" (UniqueName: \"kubernetes.io/projected/748ca8b1-9602-4a02-a162-d222afb56e5e-kube-api-access-m9vmw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:38.532638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:38.532634    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748ca8b1-9602-4a02-a162-d222afb56e5e-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:38.569819 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:53:38.569776    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:53:39.208249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:39.204648    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq" event={"ID":"748ca8b1-9602-4a02-a162-d222afb56e5e","Type":"ContainerDied","Data":"55bdd0e15fecc985e5cb19c0985f9ae03c60c0b2b7a6cf82063ceabe1ee55d85"}
Oct 30 13:53:39.208249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:39.204676    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"
Oct 30 13:53:39.208249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:39.204701    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55bdd0e15fecc985e5cb19c0985f9ae03c60c0b2b7a6cf82063ceabe1ee55d85"
Oct 30 13:53:45.358616 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.358572    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"]
Oct 30 13:53:45.359238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.359143    4993 cpu_manager.go:414] "containerMap: removing container" podUID="748ca8b1-9602-4a02-a162-d222afb56e5e" containerName="revoke-keys"
Oct 30 13:53:45.359238 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.359158    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748ca8b1-9602-4a02-a162-d222afb56e5e" containerName="revoke-keys"
Oct 30 13:53:45.359328 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.359307    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="748ca8b1-9602-4a02-a162-d222afb56e5e" containerName="revoke-keys"
Oct 30 13:53:45.363173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.363148    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.378035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.378002    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:45.389814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.389720    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:53:45.390062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.389967    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:53:45.390062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.390055    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:53:45.398046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.395619    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"]
Oct 30 13:53:45.446163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.446110    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh4dx\" (UniqueName: \"kubernetes.io/projected/a32d9c54-c294-4d00-ac1c-31829f7a67ef-kube-api-access-bh4dx\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.446413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.446291    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.446413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.446360    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.548174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.548130    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.548403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.548232    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.548403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.548394    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh4dx\" (UniqueName: \"kubernetes.io/projected/a32d9c54-c294-4d00-ac1c-31829f7a67ef-kube-api-access-bh4dx\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.549039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.549012    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.550864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.550835    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.572081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.572049    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh4dx\" (UniqueName: \"kubernetes.io/projected/a32d9c54-c294-4d00-ac1c-31829f7a67ef-kube-api-access-bh4dx\") pod \"maas-api-revoke-sub-0cbbd780-c78t5\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:45.682280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:45.681537    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:46.227268 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.227233    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"]
Oct 30 13:53:46.229117 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:53:46.229084    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32d9c54_c294_4d00_ac1c_31829f7a67ef.slice/crio-2120f93ea3109f3594a90b71c7f7867fdbdc0f5c82319244684c248ebae4a28d WatchSource:0}: Error finding container 2120f93ea3109f3594a90b71c7f7867fdbdc0f5c82319244684c248ebae4a28d: Status 404 returned error can't find the container with id 2120f93ea3109f3594a90b71c7f7867fdbdc0f5c82319244684c248ebae4a28d
Oct 30 13:53:46.278775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.273596    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5" event={"ID":"a32d9c54-c294-4d00-ac1c-31829f7a67ef","Type":"ContainerStarted","Data":"2120f93ea3109f3594a90b71c7f7867fdbdc0f5c82319244684c248ebae4a28d"}
Oct 30 13:53:46.278775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.276959    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"]
Oct 30 13:53:46.285108 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.285076    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.299697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.299665    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"]
Oct 30 13:53:46.375134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.375093    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.375134 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.375142    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.375816 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.375351    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44p9\" (UniqueName: \"kubernetes.io/projected/d923ffb8-07b8-46e2-a190-6afd33e4d52c-kube-api-access-m44p9\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.476296 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.476176    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44p9\" (UniqueName: \"kubernetes.io/projected/d923ffb8-07b8-46e2-a190-6afd33e4d52c-kube-api-access-m44p9\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.476505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.476325    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.476505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.476356    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.477188 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.477158    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.479012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.478986    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.499458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.499420    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44p9\" (UniqueName: \"kubernetes.io/projected/d923ffb8-07b8-46e2-a190-6afd33e4d52c-kube-api-access-m44p9\") pod \"maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.600537 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.600492    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:46.911851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:46.911817    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"]
Oct 30 13:53:46.913581 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:53:46.913539    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd923ffb8_07b8_46e2_a190_6afd33e4d52c.slice/crio-baefda5b533cf66c9a72e72d4c153c3bb660f7f7e275f6259965db93fdfbe6bf WatchSource:0}: Error finding container baefda5b533cf66c9a72e72d4c153c3bb660f7f7e275f6259965db93fdfbe6bf: Status 404 returned error can't find the container with id baefda5b533cf66c9a72e72d4c153c3bb660f7f7e275f6259965db93fdfbe6bf
Oct 30 13:53:47.034525 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:47.034478    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"]
Oct 30 13:53:47.038457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:47.038424    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480237d9-qhpzz"]
Oct 30 13:53:47.282710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:47.282665    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk" event={"ID":"d923ffb8-07b8-46e2-a190-6afd33e4d52c","Type":"ContainerStarted","Data":"baefda5b533cf66c9a72e72d4c153c3bb660f7f7e275f6259965db93fdfbe6bf"}
Oct 30 13:53:47.284692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:47.284661    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5" event={"ID":"a32d9c54-c294-4d00-ac1c-31829f7a67ef","Type":"ContainerStarted","Data":"ab5b8876946b45c740399b74b49181db31a8e70463b0005e589b5a53674fa94b"}
Oct 30 13:53:47.327107 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:47.327045    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5" podStartSLOduration=2.327028633 podStartE2EDuration="2.327028633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:47.32402853 +0000 UTC m=+4891.799748140" watchObservedRunningTime="2026-10-30 13:53:47.327028633 +0000 UTC m=+4891.802748244"
Oct 30 13:53:47.692590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:47.692548    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0392f69a-8db8-4cc4-8202-fe06d5d1cd08" path="/var/lib/kubelet/pods/0392f69a-8db8-4cc4-8202-fe06d5d1cd08/volumes"
Oct 30 13:53:48.294688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:48.294650    4993 generic.go:356] "Generic (PLEG): container finished" podID="d923ffb8-07b8-46e2-a190-6afd33e4d52c" containerID="fef039eb9952fc5394f8874b1473dbe4c5f3ca5b08788193bedd862959e33e39" exitCode=0
Oct 30 13:53:48.294916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:48.294742    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk" event={"ID":"d923ffb8-07b8-46e2-a190-6afd33e4d52c","Type":"ContainerDied","Data":"fef039eb9952fc5394f8874b1473dbe4c5f3ca5b08788193bedd862959e33e39"}
Oct 30 13:53:48.296543 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:48.296510    4993 generic.go:356] "Generic (PLEG): container finished" podID="a32d9c54-c294-4d00-ac1c-31829f7a67ef" containerID="ab5b8876946b45c740399b74b49181db31a8e70463b0005e589b5a53674fa94b" exitCode=0
Oct 30 13:53:48.296702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:48.296545    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5" event={"ID":"a32d9c54-c294-4d00-ac1c-31829f7a67ef","Type":"ContainerDied","Data":"ab5b8876946b45c740399b74b49181db31a8e70463b0005e589b5a53674fa94b"}
Oct 30 13:53:49.493287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.493259    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:49.521233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.521186    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32d9c54-c294-4d00-ac1c-31829f7a67ef-kube-api-access-bh4dx\" (UniqueName: \"kubernetes.io/projected/a32d9c54-c294-4d00-ac1c-31829f7a67ef-kube-api-access-bh4dx\") pod \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") "
Oct 30 13:53:49.521391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.521260    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-openshift-ca\") pod \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") "
Oct 30 13:53:49.521391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.521352    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-xks-ca\") pod \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\" (UID: \"a32d9c54-c294-4d00-ac1c-31829f7a67ef\") "
Oct 30 13:53:49.521629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.521597    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-openshift-ca" pod "a32d9c54-c294-4d00-ac1c-31829f7a67ef" (UID: "a32d9c54-c294-4d00-ac1c-31829f7a67ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:49.522071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.522043    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:49.523608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.523582    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-xks-ca" pod "a32d9c54-c294-4d00-ac1c-31829f7a67ef" (UID: "a32d9c54-c294-4d00-ac1c-31829f7a67ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:49.523744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.523720    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32d9c54-c294-4d00-ac1c-31829f7a67ef-kube-api-access-bh4dx" pod "a32d9c54-c294-4d00-ac1c-31829f7a67ef" (UID: "a32d9c54-c294-4d00-ac1c-31829f7a67ef"). InnerVolumeSpecName "kube-api-access-bh4dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:49.552691 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.552661    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:49.622626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.622540    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-openshift-ca\") pod \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") "
Oct 30 13:53:49.622813 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.622701    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d923ffb8-07b8-46e2-a190-6afd33e4d52c-kube-api-access-m44p9\" (UniqueName: \"kubernetes.io/projected/d923ffb8-07b8-46e2-a190-6afd33e4d52c-kube-api-access-m44p9\") pod \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") "
Oct 30 13:53:49.622886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.622828    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-xks-ca\") pod \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\" (UID: \"d923ffb8-07b8-46e2-a190-6afd33e4d52c\") "
Oct 30 13:53:49.622985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.622956    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-openshift-ca" pod "d923ffb8-07b8-46e2-a190-6afd33e4d52c" (UID: "d923ffb8-07b8-46e2-a190-6afd33e4d52c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:49.623302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.623276    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh4dx\" (UniqueName: \"kubernetes.io/projected/a32d9c54-c294-4d00-ac1c-31829f7a67ef-kube-api-access-bh4dx\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:49.623302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.623298    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:49.623518 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.623307    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32d9c54-c294-4d00-ac1c-31829f7a67ef-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:49.625287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.625253    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d923ffb8-07b8-46e2-a190-6afd33e4d52c-kube-api-access-m44p9" pod "d923ffb8-07b8-46e2-a190-6afd33e4d52c" (UID: "d923ffb8-07b8-46e2-a190-6afd33e4d52c"). InnerVolumeSpecName "kube-api-access-m44p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:49.625492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.625478    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-xks-ca" pod "d923ffb8-07b8-46e2-a190-6afd33e4d52c" (UID: "d923ffb8-07b8-46e2-a190-6afd33e4d52c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:49.724146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.724111    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44p9\" (UniqueName: \"kubernetes.io/projected/d923ffb8-07b8-46e2-a190-6afd33e4d52c-kube-api-access-m44p9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:49.724146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:49.724143    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d923ffb8-07b8-46e2-a190-6afd33e4d52c-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:53:50.312677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:50.312643    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"
Oct 30 13:53:50.312895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:50.312636    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk" event={"ID":"d923ffb8-07b8-46e2-a190-6afd33e4d52c","Type":"ContainerDied","Data":"baefda5b533cf66c9a72e72d4c153c3bb660f7f7e275f6259965db93fdfbe6bf"}
Oct 30 13:53:50.312895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:50.312766    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baefda5b533cf66c9a72e72d4c153c3bb660f7f7e275f6259965db93fdfbe6bf"
Oct 30 13:53:50.314138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:50.314116    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"
Oct 30 13:53:50.314321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:50.314135    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5" event={"ID":"a32d9c54-c294-4d00-ac1c-31829f7a67ef","Type":"ContainerDied","Data":"2120f93ea3109f3594a90b71c7f7867fdbdc0f5c82319244684c248ebae4a28d"}
Oct 30 13:53:50.314321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:50.314173    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2120f93ea3109f3594a90b71c7f7867fdbdc0f5c82319244684c248ebae4a28d"
Oct 30 13:53:56.308545 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:56.308508    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"]
Oct 30 13:53:59.439021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:59.438978    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"]
Oct 30 13:53:59.450772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:59.450733    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6ff0e6-7df72"]
Oct 30 13:53:59.693748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:53:59.693667    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9c4cc0-5614-4656-b93f-d93cce54da32" path="/var/lib/kubelet/pods/0b9c4cc0-5614-4656-b93f-d93cce54da32/volumes"
Oct 30 13:54:05.534682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.534638    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"]
Oct 30 13:54:05.870562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.870475    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"]
Oct 30 13:54:05.871023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.871010    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a32d9c54-c294-4d00-ac1c-31829f7a67ef" containerName="revoke-keys"
Oct 30 13:54:05.871097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.871024    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32d9c54-c294-4d00-ac1c-31829f7a67ef" containerName="revoke-keys"
Oct 30 13:54:05.871097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.871060    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d923ffb8-07b8-46e2-a190-6afd33e4d52c" containerName="revoke-keys"
Oct 30 13:54:05.871097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.871066    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d923ffb8-07b8-46e2-a190-6afd33e4d52c" containerName="revoke-keys"
Oct 30 13:54:05.871299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.871194    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32d9c54-c294-4d00-ac1c-31829f7a67ef" containerName="revoke-keys"
Oct 30 13:54:05.871299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.871236    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d923ffb8-07b8-46e2-a190-6afd33e4d52c" containerName="revoke-keys"
Oct 30 13:54:05.874922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.874885    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:05.889615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.889586    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:54:05.901832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.901785    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:54:05.902076 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.901842    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:54:05.905949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.905916    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:54:05.910248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.910183    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"]
Oct 30 13:54:05.965398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.965348    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:05.965398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.965406    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfbk\" (UniqueName: \"kubernetes.io/projected/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-kube-api-access-rgfbk\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:05.965639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:05.965511    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:06.066839 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:06.066793    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:06.066839 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:06.066852    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfbk\" (UniqueName: \"kubernetes.io/projected/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-kube-api-access-rgfbk\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:06.067160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:06.066894    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:06.067764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:06.067730    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:06.069730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:06.069638    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:06.090222 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:06.090172    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfbk\" (UniqueName: \"kubernetes.io/projected/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-kube-api-access-rgfbk\") pod \"maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:06.192132 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:06.192037    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:06.502159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:06.502124    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"]
Oct 30 13:54:06.504584 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:54:06.504556    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80c1cfa_0b3e_43e7_831b_a9ff0a45d38c.slice/crio-de55fe95305032d5be02f2267b0b4db640f1ccebb570ce54d03e3b2314bcd174 WatchSource:0}: Error finding container de55fe95305032d5be02f2267b0b4db640f1ccebb570ce54d03e3b2314bcd174: Status 404 returned error can't find the container with id de55fe95305032d5be02f2267b0b4db640f1ccebb570ce54d03e3b2314bcd174
Oct 30 13:54:07.499634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:07.499591    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl" event={"ID":"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c","Type":"ContainerStarted","Data":"aa85670f981ac2d07a9e9544ce3d3573659803e9f4187900e214e2e3da919565"}
Oct 30 13:54:07.500044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:07.499661    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl" event={"ID":"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c","Type":"ContainerStarted","Data":"de55fe95305032d5be02f2267b0b4db640f1ccebb570ce54d03e3b2314bcd174"}
Oct 30 13:54:07.542677 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:07.542609    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl" podStartSLOduration=2.542590528 podStartE2EDuration="2.542590528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:07.539335807 +0000 UTC m=+4912.015055417" watchObservedRunningTime="2026-10-30 13:54:07.542590528 +0000 UTC m=+4912.018310138"
Oct 30 13:54:08.514785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:08.514744    4993 generic.go:356] "Generic (PLEG): container finished" podID="e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c" containerID="aa85670f981ac2d07a9e9544ce3d3573659803e9f4187900e214e2e3da919565" exitCode=0
Oct 30 13:54:08.515436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:08.514843    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl" event={"ID":"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c","Type":"ContainerDied","Data":"aa85670f981ac2d07a9e9544ce3d3573659803e9f4187900e214e2e3da919565"}
Oct 30 13:54:08.582688 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:54:08.582646    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:54:09.695968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.695929    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:09.767849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.765926    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-kube-api-access-rgfbk\" (UniqueName: \"kubernetes.io/projected/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-kube-api-access-rgfbk\") pod \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") "
Oct 30 13:54:09.768556 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.768526    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-xks-ca\") pod \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") "
Oct 30 13:54:09.768736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.768691    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-openshift-ca\") pod \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\" (UID: \"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c\") "
Oct 30 13:54:09.770233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.770187    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-openshift-ca" pod "e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c" (UID: "e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:09.771724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.771692    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-xks-ca" pod "e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c" (UID: "e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:09.771933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.771909    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-kube-api-access-rgfbk" pod "e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c" (UID: "e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c"). InnerVolumeSpecName "kube-api-access-rgfbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:09.870860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.870818    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.870860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.870856    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.870860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:09.870867    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgfbk\" (UniqueName: \"kubernetes.io/projected/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c-kube-api-access-rgfbk\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:10.531455 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:10.531420    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"
Oct 30 13:54:10.531650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:10.531449    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl" event={"ID":"e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c","Type":"ContainerDied","Data":"de55fe95305032d5be02f2267b0b4db640f1ccebb570ce54d03e3b2314bcd174"}
Oct 30 13:54:10.531650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:10.531527    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de55fe95305032d5be02f2267b0b4db640f1ccebb570ce54d03e3b2314bcd174"
Oct 30 13:54:11.332807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.332738    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm" podUID="cbc29550-dc74-4259-8bba-5d6115bfb864" containerName="main" containerID="cri-o://e448ffa20a7ecd83211f7cda1c9505b82753122c85b078ad72e6f467e17bedc2" gracePeriod=45
Oct 30 13:54:11.549629 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.549585    4993 generic.go:356] "Generic (PLEG): container finished" podID="cbc29550-dc74-4259-8bba-5d6115bfb864" containerID="e448ffa20a7ecd83211f7cda1c9505b82753122c85b078ad72e6f467e17bedc2" exitCode=0
Oct 30 13:54:11.549877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.549851    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm" event={"ID":"cbc29550-dc74-4259-8bba-5d6115bfb864","Type":"ContainerDied","Data":"e448ffa20a7ecd83211f7cda1c9505b82753122c85b078ad72e6f467e17bedc2"}
Oct 30 13:54:11.740059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.740014    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:54:11.817252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817189    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-dshm\") pod \"cbc29550-dc74-4259-8bba-5d6115bfb864\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") "
Oct 30 13:54:11.817481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817368    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc29550-dc74-4259-8bba-5d6115bfb864-kube-api-access-tsqpp\" (UniqueName: \"kubernetes.io/projected/cbc29550-dc74-4259-8bba-5d6115bfb864-kube-api-access-tsqpp\") pod \"cbc29550-dc74-4259-8bba-5d6115bfb864\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") "
Oct 30 13:54:11.817481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817419    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-tmp-dir\") pod \"cbc29550-dc74-4259-8bba-5d6115bfb864\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") "
Oct 30 13:54:11.817481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817454    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-model-cache\") pod \"cbc29550-dc74-4259-8bba-5d6115bfb864\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") "
Oct 30 13:54:11.817686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817490    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc29550-dc74-4259-8bba-5d6115bfb864-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc29550-dc74-4259-8bba-5d6115bfb864-tls-certs\") pod \"cbc29550-dc74-4259-8bba-5d6115bfb864\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") "
Oct 30 13:54:11.817686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817584    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-home\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-home\") pod \"cbc29550-dc74-4259-8bba-5d6115bfb864\" (UID: \"cbc29550-dc74-4259-8bba-5d6115bfb864\") "
Oct 30 13:54:11.817799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817768    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-model-cache" pod "cbc29550-dc74-4259-8bba-5d6115bfb864" (UID: "cbc29550-dc74-4259-8bba-5d6115bfb864"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:11.817799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817783    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-tmp-dir" pod "cbc29550-dc74-4259-8bba-5d6115bfb864" (UID: "cbc29550-dc74-4259-8bba-5d6115bfb864"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:11.817934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.817908    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-home" pod "cbc29550-dc74-4259-8bba-5d6115bfb864" (UID: "cbc29550-dc74-4259-8bba-5d6115bfb864"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:11.818286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.818262    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:11.818395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.818288    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:11.818395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.818305    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:11.819985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.819959    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc29550-dc74-4259-8bba-5d6115bfb864-kube-api-access-tsqpp" pod "cbc29550-dc74-4259-8bba-5d6115bfb864" (UID: "cbc29550-dc74-4259-8bba-5d6115bfb864"). InnerVolumeSpecName "kube-api-access-tsqpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:11.820119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.820089    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-dshm" pod "cbc29550-dc74-4259-8bba-5d6115bfb864" (UID: "cbc29550-dc74-4259-8bba-5d6115bfb864"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:11.820286 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.820263    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc29550-dc74-4259-8bba-5d6115bfb864-tls-certs" pod "cbc29550-dc74-4259-8bba-5d6115bfb864" (UID: "cbc29550-dc74-4259-8bba-5d6115bfb864"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:11.920040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.919932    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc29550-dc74-4259-8bba-5d6115bfb864-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:11.920040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.919984    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc29550-dc74-4259-8bba-5d6115bfb864-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:11.920040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:11.919994    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqpp\" (UniqueName: \"kubernetes.io/projected/cbc29550-dc74-4259-8bba-5d6115bfb864-kube-api-access-tsqpp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:12.562550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.562516    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"
Oct 30 13:54:12.562550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.562533    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm" event={"ID":"cbc29550-dc74-4259-8bba-5d6115bfb864","Type":"ContainerDied","Data":"819b370022609b607d4f70215c9920af4e5d0c02e1c5131b64a285ab72a06adc"}
Oct 30 13:54:12.563143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.562590    4993 scope.go:122] "RemoveContainer" containerID="e448ffa20a7ecd83211f7cda1c9505b82753122c85b078ad72e6f467e17bedc2"
Oct 30 13:54:12.585152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.585114    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"]
Oct 30 13:54:12.585947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.585917    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cbc29550-dc74-4259-8bba-5d6115bfb864" containerName="main"
Oct 30 13:54:12.586045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.585950    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc29550-dc74-4259-8bba-5d6115bfb864" containerName="main"
Oct 30 13:54:12.586045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.585977    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c" containerName="revoke-keys"
Oct 30 13:54:12.586045 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.585985    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c" containerName="revoke-keys"
Oct 30 13:54:12.586327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.586181    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc29550-dc74-4259-8bba-5d6115bfb864" containerName="main"
Oct 30 13:54:12.586327 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.586198    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c" containerName="revoke-keys"
Oct 30 13:54:12.591307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.591281    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.607025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.606995    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:54:12.607025 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.606995    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:54:12.612092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.612057    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"]
Oct 30 13:54:12.618275 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.618240    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"]
Oct 30 13:54:12.626250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.626194    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e9cb4a/sub-test-model-e9cb4a-kserve-565889df4f-z76tm"]
Oct 30 13:54:12.638571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.638533    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:54:12.638775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.638747    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:54:12.730499 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.730463    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.730724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.730596    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.730821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.730793    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhk4l\" (UniqueName: \"kubernetes.io/projected/06f6482f-5cc0-44d7-93d0-7e01944205b7-kube-api-access-fhk4l\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.831840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.831732    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.832039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.831860    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhk4l\" (UniqueName: \"kubernetes.io/projected/06f6482f-5cc0-44d7-93d0-7e01944205b7-kube-api-access-fhk4l\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.832039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.832006    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.832760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.832729    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.834331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.834297    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.855877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.855837    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhk4l\" (UniqueName: \"kubernetes.io/projected/06f6482f-5cc0-44d7-93d0-7e01944205b7-kube-api-access-fhk4l\") pod \"maas-api-revoke-sub-0bd9da67-cshsq\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:12.908429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:12.908388    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:13.225219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:13.225172    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"]
Oct 30 13:54:13.226940 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:54:13.226905    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f6482f_5cc0_44d7_93d0_7e01944205b7.slice/crio-78c395953bc58d270c0cca66bd588a8fff289780c23087d24139919b0e9098e8 WatchSource:0}: Error finding container 78c395953bc58d270c0cca66bd588a8fff289780c23087d24139919b0e9098e8: Status 404 returned error can't find the container with id 78c395953bc58d270c0cca66bd588a8fff289780c23087d24139919b0e9098e8
Oct 30 13:54:13.574769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:13.574724    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq" event={"ID":"06f6482f-5cc0-44d7-93d0-7e01944205b7","Type":"ContainerStarted","Data":"78c395953bc58d270c0cca66bd588a8fff289780c23087d24139919b0e9098e8"}
Oct 30 13:54:13.708439 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:13.708402    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc29550-dc74-4259-8bba-5d6115bfb864" path="/var/lib/kubelet/pods/cbc29550-dc74-4259-8bba-5d6115bfb864/volumes"
Oct 30 13:54:14.587459 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:14.587415    4993 generic.go:356] "Generic (PLEG): container finished" podID="06f6482f-5cc0-44d7-93d0-7e01944205b7" containerID="c848b78691f1ccc9bc193ad265c291b152598b83b5db8ed39778c7c47ca1d2ad" exitCode=0
Oct 30 13:54:14.587912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:14.587502    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq" event={"ID":"06f6482f-5cc0-44d7-93d0-7e01944205b7","Type":"ContainerDied","Data":"c848b78691f1ccc9bc193ad265c291b152598b83b5db8ed39778c7c47ca1d2ad"}
Oct 30 13:54:15.768780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.768747    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:15.877123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.877024    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-xks-ca\") pod \"06f6482f-5cc0-44d7-93d0-7e01944205b7\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") "
Oct 30 13:54:15.877349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.877135    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-openshift-ca\") pod \"06f6482f-5cc0-44d7-93d0-7e01944205b7\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") "
Oct 30 13:54:15.877349 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.877201    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f6482f-5cc0-44d7-93d0-7e01944205b7-kube-api-access-fhk4l\" (UniqueName: \"kubernetes.io/projected/06f6482f-5cc0-44d7-93d0-7e01944205b7-kube-api-access-fhk4l\") pod \"06f6482f-5cc0-44d7-93d0-7e01944205b7\" (UID: \"06f6482f-5cc0-44d7-93d0-7e01944205b7\") "
Oct 30 13:54:15.877581 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.877554    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-openshift-ca" pod "06f6482f-5cc0-44d7-93d0-7e01944205b7" (UID: "06f6482f-5cc0-44d7-93d0-7e01944205b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:15.877866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.877844    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:15.879536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.879512    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-xks-ca" pod "06f6482f-5cc0-44d7-93d0-7e01944205b7" (UID: "06f6482f-5cc0-44d7-93d0-7e01944205b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:15.879664 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.879643    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f6482f-5cc0-44d7-93d0-7e01944205b7-kube-api-access-fhk4l" pod "06f6482f-5cc0-44d7-93d0-7e01944205b7" (UID: "06f6482f-5cc0-44d7-93d0-7e01944205b7"). InnerVolumeSpecName "kube-api-access-fhk4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:15.979053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.979010    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhk4l\" (UniqueName: \"kubernetes.io/projected/06f6482f-5cc0-44d7-93d0-7e01944205b7-kube-api-access-fhk4l\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:15.979053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:15.979048    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f6482f-5cc0-44d7-93d0-7e01944205b7-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:16.607717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:16.607683    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq" event={"ID":"06f6482f-5cc0-44d7-93d0-7e01944205b7","Type":"ContainerDied","Data":"78c395953bc58d270c0cca66bd588a8fff289780c23087d24139919b0e9098e8"}
Oct 30 13:54:16.608003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:16.607720    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"
Oct 30 13:54:16.608003 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:16.607732    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78c395953bc58d270c0cca66bd588a8fff289780c23087d24139919b0e9098e8"
Oct 30 13:54:20.563608 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.563546    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp" podUID="82dab4c8-de9b-46dd-812d-40f6c8ded436" containerName="main" containerID="cri-o://d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a" gracePeriod=45
Oct 30 13:54:20.625095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.625054    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"]
Oct 30 13:54:20.668234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.662988    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-88e9f3-jvdpk"]
Oct 30 13:54:20.733621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.733300    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp" podUID="82dab4c8-de9b-46dd-812d-40f6c8ded436" containerName="main" probeResult="failure" output="Get \"https://10.132.1.107:8000/ready\": dial tcp 10.132.1.107:8000: connect: connection refused"
Oct 30 13:54:20.881646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.881607    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:54:20.889810 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.889776    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-home\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-home\") pod \"82dab4c8-de9b-46dd-812d-40f6c8ded436\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") "
Oct 30 13:54:20.890019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.889839    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82dab4c8-de9b-46dd-812d-40f6c8ded436-tls-certs\" (UniqueName: \"kubernetes.io/secret/82dab4c8-de9b-46dd-812d-40f6c8ded436-tls-certs\") pod \"82dab4c8-de9b-46dd-812d-40f6c8ded436\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") "
Oct 30 13:54:20.890019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.889881    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-model-cache\") pod \"82dab4c8-de9b-46dd-812d-40f6c8ded436\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") "
Oct 30 13:54:20.890019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890012    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-tmp-dir\") pod \"82dab4c8-de9b-46dd-812d-40f6c8ded436\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") "
Oct 30 13:54:20.890201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890066    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-home" pod "82dab4c8-de9b-46dd-812d-40f6c8ded436" (UID: "82dab4c8-de9b-46dd-812d-40f6c8ded436"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:20.890201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890178    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dab4c8-de9b-46dd-812d-40f6c8ded436-kube-api-access-ssl26\" (UniqueName: \"kubernetes.io/projected/82dab4c8-de9b-46dd-812d-40f6c8ded436-kube-api-access-ssl26\") pod \"82dab4c8-de9b-46dd-812d-40f6c8ded436\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") "
Oct 30 13:54:20.890201 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890190    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-model-cache" pod "82dab4c8-de9b-46dd-812d-40f6c8ded436" (UID: "82dab4c8-de9b-46dd-812d-40f6c8ded436"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:20.890382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890227    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-dshm\") pod \"82dab4c8-de9b-46dd-812d-40f6c8ded436\" (UID: \"82dab4c8-de9b-46dd-812d-40f6c8ded436\") "
Oct 30 13:54:20.890382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890317    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-tmp-dir" pod "82dab4c8-de9b-46dd-812d-40f6c8ded436" (UID: "82dab4c8-de9b-46dd-812d-40f6c8ded436"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:20.890724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890699    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:20.890724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890723    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:20.890943 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.890739    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:20.892258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.892194    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dab4c8-de9b-46dd-812d-40f6c8ded436-tls-certs" pod "82dab4c8-de9b-46dd-812d-40f6c8ded436" (UID: "82dab4c8-de9b-46dd-812d-40f6c8ded436"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:20.892399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.892373    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-dshm" pod "82dab4c8-de9b-46dd-812d-40f6c8ded436" (UID: "82dab4c8-de9b-46dd-812d-40f6c8ded436"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:20.892661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.892640    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dab4c8-de9b-46dd-812d-40f6c8ded436-kube-api-access-ssl26" pod "82dab4c8-de9b-46dd-812d-40f6c8ded436" (UID: "82dab4c8-de9b-46dd-812d-40f6c8ded436"). InnerVolumeSpecName "kube-api-access-ssl26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:20.991457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.991417    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssl26\" (UniqueName: \"kubernetes.io/projected/82dab4c8-de9b-46dd-812d-40f6c8ded436-kube-api-access-ssl26\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:20.991457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.991451    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82dab4c8-de9b-46dd-812d-40f6c8ded436-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:20.991457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:20.991463    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82dab4c8-de9b-46dd-812d-40f6c8ded436-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:21.686337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.686300    4993 generic.go:356] "Generic (PLEG): container finished" podID="82dab4c8-de9b-46dd-812d-40f6c8ded436" containerID="d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a" exitCode=0
Oct 30 13:54:21.686762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.686428    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"
Oct 30 13:54:21.694477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.694449    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d923ffb8-07b8-46e2-a190-6afd33e4d52c" path="/var/lib/kubelet/pods/d923ffb8-07b8-46e2-a190-6afd33e4d52c/volumes"
Oct 30 13:54:21.695060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.695040    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp" event={"ID":"82dab4c8-de9b-46dd-812d-40f6c8ded436","Type":"ContainerDied","Data":"d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a"}
Oct 30 13:54:21.695116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.695074    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp" event={"ID":"82dab4c8-de9b-46dd-812d-40f6c8ded436","Type":"ContainerDied","Data":"c314ea748674fde7b4134aa1597f51139de978b3dd7c0b6e8441f46967a55f3f"}
Oct 30 13:54:21.695116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.695102    4993 scope.go:122] "RemoveContainer" containerID="d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a"
Oct 30 13:54:21.708200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.708176    4993 scope.go:122] "RemoveContainer" containerID="d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a"
Oct 30 13:54:21.708515 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:54:21.708491    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a\": container with ID starting with d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a not found: ID does not exist" containerID="d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a"
Oct 30 13:54:21.708571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.708526    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a"} err="failed to get container status \"d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a\": rpc error: code = NotFound desc = could not find container \"d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a\": container with ID starting with d40b389ae792042eb0eef377b818b586c3689df2415709fef8b89ff645865a4a not found: ID does not exist"
Oct 30 13:54:21.741465 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.741426    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"]
Oct 30 13:54:21.753804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:21.753744    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-92855c/sub-test-model-92855c-kserve-57844fd985-65xhp"]
Oct 30 13:54:22.741920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.741779    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"]
Oct 30 13:54:22.749249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.749196    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-63499e-l9gjl"]
Oct 30 13:54:22.924221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.924177    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"]
Oct 30 13:54:22.924839 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.924823    4993 cpu_manager.go:414] "containerMap: removing container" podUID="82dab4c8-de9b-46dd-812d-40f6c8ded436" containerName="main"
Oct 30 13:54:22.924890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.924841    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dab4c8-de9b-46dd-812d-40f6c8ded436" containerName="main"
Oct 30 13:54:22.924890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.924868    4993 cpu_manager.go:414] "containerMap: removing container" podUID="06f6482f-5cc0-44d7-93d0-7e01944205b7" containerName="revoke-keys"
Oct 30 13:54:22.924890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.924873    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f6482f-5cc0-44d7-93d0-7e01944205b7" containerName="revoke-keys"
Oct 30 13:54:22.925001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.924991    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dab4c8-de9b-46dd-812d-40f6c8ded436" containerName="main"
Oct 30 13:54:22.925053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.925003    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f6482f-5cc0-44d7-93d0-7e01944205b7" containerName="revoke-keys"
Oct 30 13:54:22.929696 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.929675    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:22.945426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.945396    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:54:22.945661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.945642    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:54:22.946026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.946002    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:54:22.946144 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.946124    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:54:22.950604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:22.950547    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"]
Oct 30 13:54:23.046429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.046321    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8fd9\" (UniqueName: \"kubernetes.io/projected/a4aa7a8b-d07b-4035-88d1-16a9dd702412-kube-api-access-d8fd9\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.046635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.046529    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.046771 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.046749    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.148403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.148365    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.148589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.148512    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.148589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.148563    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8fd9\" (UniqueName: \"kubernetes.io/projected/a4aa7a8b-d07b-4035-88d1-16a9dd702412-kube-api-access-d8fd9\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.149192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.149158    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.151108 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.151085    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.172785 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.172735    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8fd9\" (UniqueName: \"kubernetes.io/projected/a4aa7a8b-d07b-4035-88d1-16a9dd702412-kube-api-access-d8fd9\") pod \"maas-api-revoke-sub-34271737-w8rtf\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.244588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.244545    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:23.556244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.556170    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"]
Oct 30 13:54:23.557808 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:54:23.557781    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4aa7a8b_d07b_4035_88d1_16a9dd702412.slice/crio-62c6003810c4d78f6e92beb13eb50a8c2c06678f353696780224aaef0ebb60e5 WatchSource:0}: Error finding container 62c6003810c4d78f6e92beb13eb50a8c2c06678f353696780224aaef0ebb60e5: Status 404 returned error can't find the container with id 62c6003810c4d78f6e92beb13eb50a8c2c06678f353696780224aaef0ebb60e5
Oct 30 13:54:23.693080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.693040    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dab4c8-de9b-46dd-812d-40f6c8ded436" path="/var/lib/kubelet/pods/82dab4c8-de9b-46dd-812d-40f6c8ded436/volumes"
Oct 30 13:54:23.693699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.693676    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c" path="/var/lib/kubelet/pods/e80c1cfa-0b3e-43e7-831b-a9ff0a45d38c/volumes"
Oct 30 13:54:23.733988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:23.733947    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf" event={"ID":"a4aa7a8b-d07b-4035-88d1-16a9dd702412","Type":"ContainerStarted","Data":"62c6003810c4d78f6e92beb13eb50a8c2c06678f353696780224aaef0ebb60e5"}
Oct 30 13:54:24.745536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:24.745498    4993 generic.go:356] "Generic (PLEG): container finished" podID="a4aa7a8b-d07b-4035-88d1-16a9dd702412" containerID="99325c2879948b6f89d570d50be2e786c9b017de336bf156f50b765916b24adb" exitCode=0
Oct 30 13:54:24.745968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:24.745529    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf" event={"ID":"a4aa7a8b-d07b-4035-88d1-16a9dd702412","Type":"ContainerDied","Data":"99325c2879948b6f89d570d50be2e786c9b017de336bf156f50b765916b24adb"}
Oct 30 13:54:25.923035 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:25.922997    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:25.990633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:25.990588    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-xks-ca\") pod \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") "
Oct 30 13:54:25.990832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:25.990663    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-openshift-ca\") pod \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") "
Oct 30 13:54:25.990832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:25.990707    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4aa7a8b-d07b-4035-88d1-16a9dd702412-kube-api-access-d8fd9\" (UniqueName: \"kubernetes.io/projected/a4aa7a8b-d07b-4035-88d1-16a9dd702412-kube-api-access-d8fd9\") pod \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\" (UID: \"a4aa7a8b-d07b-4035-88d1-16a9dd702412\") "
Oct 30 13:54:25.991135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:25.991108    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-openshift-ca" pod "a4aa7a8b-d07b-4035-88d1-16a9dd702412" (UID: "a4aa7a8b-d07b-4035-88d1-16a9dd702412"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:25.991528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:25.991504    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:25.993900 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:25.993868    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4aa7a8b-d07b-4035-88d1-16a9dd702412-kube-api-access-d8fd9" pod "a4aa7a8b-d07b-4035-88d1-16a9dd702412" (UID: "a4aa7a8b-d07b-4035-88d1-16a9dd702412"). InnerVolumeSpecName "kube-api-access-d8fd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:25.994163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:25.994143    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-xks-ca" pod "a4aa7a8b-d07b-4035-88d1-16a9dd702412" (UID: "a4aa7a8b-d07b-4035-88d1-16a9dd702412"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:26.092913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:26.092815    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8fd9\" (UniqueName: \"kubernetes.io/projected/a4aa7a8b-d07b-4035-88d1-16a9dd702412-kube-api-access-d8fd9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:26.092913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:26.092856    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4aa7a8b-d07b-4035-88d1-16a9dd702412-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:26.765480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:26.765445    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf"
Oct 30 13:54:26.765480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:26.765467    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34271737-w8rtf" event={"ID":"a4aa7a8b-d07b-4035-88d1-16a9dd702412","Type":"ContainerDied","Data":"62c6003810c4d78f6e92beb13eb50a8c2c06678f353696780224aaef0ebb60e5"}
Oct 30 13:54:26.765774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:26.765519    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62c6003810c4d78f6e92beb13eb50a8c2c06678f353696780224aaef0ebb60e5"
Oct 30 13:54:28.700902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:28.700857    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"]
Oct 30 13:54:28.701461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:28.701227    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy" containerID="cri-o://7a82797ed5dd41b241b84079f3c334b508016c99de869b2df5aab21ed7a9cdbb" gracePeriod=30
Oct 30 13:54:29.282361 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.282319    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"]
Oct 30 13:54:29.283199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.283174    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a4aa7a8b-d07b-4035-88d1-16a9dd702412" containerName="revoke-keys"
Oct 30 13:54:29.283335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.283200    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4aa7a8b-d07b-4035-88d1-16a9dd702412" containerName="revoke-keys"
Oct 30 13:54:29.283377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.283360    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4aa7a8b-d07b-4035-88d1-16a9dd702412" containerName="revoke-keys"
Oct 30 13:54:29.287081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.287063    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.306553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.306520    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"]
Oct 30 13:54:29.313821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.313797    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:54:29.313821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.313798    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:54:29.363553 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.363514    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.363786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.363574    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshg7\" (UniqueName: \"kubernetes.io/projected/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-kube-api-access-jshg7\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.363786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.363746    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.369822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.369789    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:54:29.370004 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.369878    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:54:29.465146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.465092    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.465391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.465166    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshg7\" (UniqueName: \"kubernetes.io/projected/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-kube-api-access-jshg7\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.465391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.465335    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.466090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.466067    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.467829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.467797    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.488610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.488578    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshg7\" (UniqueName: \"kubernetes.io/projected/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-kube-api-access-jshg7\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.601649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.601541    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:29.907998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:29.907956    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"]
Oct 30 13:54:29.908545 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:54:29.908519    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1ac26e_e2e8_4701_8fff_6d7dbca89df3.slice/crio-5bf7c1cf515b0d8c6229198d9ce9ac91bfa3db6251bdecc2702357497665feab WatchSource:0}: Error finding container 5bf7c1cf515b0d8c6229198d9ce9ac91bfa3db6251bdecc2702357497665feab: Status 404 returned error can't find the container with id 5bf7c1cf515b0d8c6229198d9ce9ac91bfa3db6251bdecc2702357497665feab
Oct 30 13:54:30.866006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:30.865968    4993 generic.go:356] "Generic (PLEG): container finished" podID="9d1ac26e-e2e8-4701-8fff-6d7dbca89df3" containerID="405768af9cc7bd988ad2931178cb56788db744833616db6b742e3f85a7e2dbbf" exitCode=0
Oct 30 13:54:30.866257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:30.866055    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8" event={"ID":"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3","Type":"ContainerDied","Data":"405768af9cc7bd988ad2931178cb56788db744833616db6b742e3f85a7e2dbbf"}
Oct 30 13:54:30.866257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:30.866125    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8" event={"ID":"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3","Type":"ContainerStarted","Data":"5bf7c1cf515b0d8c6229198d9ce9ac91bfa3db6251bdecc2702357497665feab"}
Oct 30 13:54:31.031260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.031151    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"]
Oct 30 13:54:31.031780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.031727    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy" containerID="cri-o://e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019" gracePeriod=30
Oct 30 13:54:31.353603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.353514    4993 patch_prober.go:29] interesting pod/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:54:31.353603 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.353573    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:54:31.612223 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.612109    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"]
Oct 30 13:54:31.616163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.616131    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.638323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.638138    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"]
Oct 30 13:54:31.721313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.721275    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.721530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.721421    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9cck\" (UniqueName: \"kubernetes.io/projected/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-kube-api-access-t9cck\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.721530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.721483    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.823097 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.823052    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.823320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.823172    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cck\" (UniqueName: \"kubernetes.io/projected/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-kube-api-access-t9cck\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.823320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.823195    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.823880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.823851    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.825807 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.825782    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.846833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.846794    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9cck\" (UniqueName: \"kubernetes.io/projected/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-kube-api-access-t9cck\") pod \"maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:31.929877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:31.929835    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:32.052613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.052578    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:32.056673 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.056633    4993 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:54:32.056837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.056686    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:54:32.126959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.126915    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-xks-ca\") pod \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") "
Oct 30 13:54:32.127177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.127112    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-openshift-ca\") pod \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") "
Oct 30 13:54:32.127352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.127330    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-kube-api-access-jshg7\" (UniqueName: \"kubernetes.io/projected/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-kube-api-access-jshg7\") pod \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\" (UID: \"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3\") "
Oct 30 13:54:32.127619 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.127522    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-openshift-ca" pod "9d1ac26e-e2e8-4701-8fff-6d7dbca89df3" (UID: "9d1ac26e-e2e8-4701-8fff-6d7dbca89df3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:32.128251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.128224    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:32.129804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.129780    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-xks-ca" pod "9d1ac26e-e2e8-4701-8fff-6d7dbca89df3" (UID: "9d1ac26e-e2e8-4701-8fff-6d7dbca89df3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:32.129885 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.129797    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-kube-api-access-jshg7" pod "9d1ac26e-e2e8-4701-8fff-6d7dbca89df3" (UID: "9d1ac26e-e2e8-4701-8fff-6d7dbca89df3"). InnerVolumeSpecName "kube-api-access-jshg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:32.229618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.229580    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshg7\" (UniqueName: \"kubernetes.io/projected/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-kube-api-access-jshg7\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:32.229618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.229616    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:32.263390 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:54:32.263340    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674223a1_9a7f_4ea8_a6de_2f5850d3a14a.slice/crio-8363413bc769b0359827c9a45ee6e3b12e7f0b6481e7de43ee12f8a9a5383085 WatchSource:0}: Error finding container 8363413bc769b0359827c9a45ee6e3b12e7f0b6481e7de43ee12f8a9a5383085: Status 404 returned error can't find the container with id 8363413bc769b0359827c9a45ee6e3b12e7f0b6481e7de43ee12f8a9a5383085
Oct 30 13:54:32.263984 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.263898    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"]
Oct 30 13:54:32.888665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.888622    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx" event={"ID":"674223a1-9a7f-4ea8-a6de-2f5850d3a14a","Type":"ContainerStarted","Data":"386cbe1722c60d32081982e924480c4c9763463dfa9026be25a4ce90049e33c8"}
Oct 30 13:54:32.888847 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.888693    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx" event={"ID":"674223a1-9a7f-4ea8-a6de-2f5850d3a14a","Type":"ContainerStarted","Data":"8363413bc769b0359827c9a45ee6e3b12e7f0b6481e7de43ee12f8a9a5383085"}
Oct 30 13:54:32.890679 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.890642    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"
Oct 30 13:54:32.890848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.890699    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8" event={"ID":"9d1ac26e-e2e8-4701-8fff-6d7dbca89df3","Type":"ContainerDied","Data":"5bf7c1cf515b0d8c6229198d9ce9ac91bfa3db6251bdecc2702357497665feab"}
Oct 30 13:54:32.890848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.890753    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf7c1cf515b0d8c6229198d9ce9ac91bfa3db6251bdecc2702357497665feab"
Oct 30 13:54:32.930530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:32.930463    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx" podStartSLOduration=1.9304454739999999 podStartE2EDuration="1.930445474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:32.928399375 +0000 UTC m=+4937.404118985" watchObservedRunningTime="2026-10-30 13:54:32.930445474 +0000 UTC m=+4937.406165109"
Oct 30 13:54:33.905359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:33.904559    4993 generic.go:356] "Generic (PLEG): container finished" podID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerID="7a82797ed5dd41b241b84079f3c334b508016c99de869b2df5aab21ed7a9cdbb" exitCode=0
Oct 30 13:54:33.905359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:33.904683    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" event={"ID":"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9","Type":"ContainerDied","Data":"7a82797ed5dd41b241b84079f3c334b508016c99de869b2df5aab21ed7a9cdbb"}
Oct 30 13:54:33.905359 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:33.904735    4993 scope.go:122] "RemoveContainer" containerID="13ddf3162ee0f030f1d304ba9802236862261cbe9e0f361ab2ba7d6f284dd9b6"
Oct 30 13:54:33.907988 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:33.907940    4993 generic.go:356] "Generic (PLEG): container finished" podID="674223a1-9a7f-4ea8-a6de-2f5850d3a14a" containerID="386cbe1722c60d32081982e924480c4c9763463dfa9026be25a4ce90049e33c8" exitCode=0
Oct 30 13:54:33.908111 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:33.908036    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx" event={"ID":"674223a1-9a7f-4ea8-a6de-2f5850d3a14a","Type":"ContainerDied","Data":"386cbe1722c60d32081982e924480c4c9763463dfa9026be25a4ce90049e33c8"}
Oct 30 13:54:34.117462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.117425    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:54:34.271181 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271124    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-token\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-token\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.271429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271201    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-data\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.271429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271344    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-envoy\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.271429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271412    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istiod-ca-cert\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.271646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271500    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-podinfo\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.271646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271602    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-kube-api-access-wtbjz\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-kube-api-access-wtbjz\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.271756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271683    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-credential-socket\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.271756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271706    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-socket\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.271871 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.271761    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-certs\") pod \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\" (UID: \"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9\") "
Oct 30 13:54:34.280232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.272579    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istiod-ca-cert" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:34.280232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.274162    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-socket" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:34.280232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.274263    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-credential-socket" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:34.280232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.274613    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-data" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:34.280232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.275317    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-certs" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:34.293986 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.293825    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-envoy" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:34.306409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.296127    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-kube-api-access-wtbjz" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "kube-api-access-wtbjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:34.306409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.300299    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-token" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:34.306409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.300450    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-podinfo" pod "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" (UID: "f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:54:34.373683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373591    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.373683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373633    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.373683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373650    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.373683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373664    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.373683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373678    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.374121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373691    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbjz\" (UniqueName: \"kubernetes.io/projected/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-kube-api-access-wtbjz\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.374121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373707    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.374121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373720    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.374121 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.373734    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:34.918432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.918400    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"
Oct 30 13:54:34.918898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.918431    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng" event={"ID":"f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9","Type":"ContainerDied","Data":"4016ad9d213578095d51bb68aa0adf59961d1910dfd31facf782fc30ea87ee81"}
Oct 30 13:54:34.918898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.918492    4993 scope.go:122] "RemoveContainer" containerID="7a82797ed5dd41b241b84079f3c334b508016c99de869b2df5aab21ed7a9cdbb"
Oct 30 13:54:34.987008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.986959    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"]
Oct 30 13:54:34.997751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:34.997717    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-88e9f3-openshift-default-68c959b4cc-xbxng"]
Oct 30 13:54:35.119595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.119543    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:35.302415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.302364    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-openshift-ca\") pod \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") "
Oct 30 13:54:35.302590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.302498    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-xks-ca\") pod \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") "
Oct 30 13:54:35.302590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.302543    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-kube-api-access-t9cck\" (UniqueName: \"kubernetes.io/projected/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-kube-api-access-t9cck\") pod \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\" (UID: \"674223a1-9a7f-4ea8-a6de-2f5850d3a14a\") "
Oct 30 13:54:35.302852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.302816    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-openshift-ca" pod "674223a1-9a7f-4ea8-a6de-2f5850d3a14a" (UID: "674223a1-9a7f-4ea8-a6de-2f5850d3a14a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:35.303218 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.303181    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:35.306321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.305225    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-kube-api-access-t9cck" pod "674223a1-9a7f-4ea8-a6de-2f5850d3a14a" (UID: "674223a1-9a7f-4ea8-a6de-2f5850d3a14a"). InnerVolumeSpecName "kube-api-access-t9cck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:35.306321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.305717    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-xks-ca" pod "674223a1-9a7f-4ea8-a6de-2f5850d3a14a" (UID: "674223a1-9a7f-4ea8-a6de-2f5850d3a14a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:35.404378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.404328    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:35.404378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.404370    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9cck\" (UniqueName: \"kubernetes.io/projected/674223a1-9a7f-4ea8-a6de-2f5850d3a14a-kube-api-access-t9cck\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:35.691601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.691520    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" path="/var/lib/kubelet/pods/f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9/volumes"
Oct 30 13:54:35.929012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.928972    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"
Oct 30 13:54:35.929509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.928987    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx" event={"ID":"674223a1-9a7f-4ea8-a6de-2f5850d3a14a","Type":"ContainerDied","Data":"8363413bc769b0359827c9a45ee6e3b12e7f0b6481e7de43ee12f8a9a5383085"}
Oct 30 13:54:35.929509 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:35.929085    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8363413bc769b0359827c9a45ee6e3b12e7f0b6481e7de43ee12f8a9a5383085"
Oct 30 13:54:36.331995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.331968    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:54:36.416484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.416441    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-socket\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.416688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.416530    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-credential-socket\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.416688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.416569    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4476629-9b97-427c-82df-d7b54567b2c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4476629-9b97-427c-82df-d7b54567b2c8-istiod-ca-cert\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.416688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.416609    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4476629-9b97-427c-82df-d7b54567b2c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4476629-9b97-427c-82df-d7b54567b2c8-istio-podinfo\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.416872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.416796    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-socket" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:36.416872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.416806    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-certs\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.416989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.416881    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-credential-socket" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:36.417047 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417005    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-kube-api-access-2842t\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-kube-api-access-2842t\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.417109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417047    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4476629-9b97-427c-82df-d7b54567b2c8-istiod-ca-cert" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:36.417109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417072    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-certs" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:36.417109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417087    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-data\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.417322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417174    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-envoy\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.417322 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417231    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-istio-token\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-istio-token\") pod \"e4476629-9b97-427c-82df-d7b54567b2c8\" (UID: \"e4476629-9b97-427c-82df-d7b54567b2c8\") "
Oct 30 13:54:36.417415 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417322    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-data" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:36.417752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417726    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.417752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417754    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.417949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417766    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.417949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417778    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.417949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.417795    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4476629-9b97-427c-82df-d7b54567b2c8-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.419387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.419360    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4476629-9b97-427c-82df-d7b54567b2c8-istio-podinfo" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:54:36.419985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.419960    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-istio-token" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:36.420099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.419994    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-kube-api-access-2842t" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "kube-api-access-2842t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:36.420172 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.420145    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-envoy" pod "e4476629-9b97-427c-82df-d7b54567b2c8" (UID: "e4476629-9b97-427c-82df-d7b54567b2c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:36.519488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.519442    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2842t\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-kube-api-access-2842t\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.519488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.519477    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4476629-9b97-427c-82df-d7b54567b2c8-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.519488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.519487    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4476629-9b97-427c-82df-d7b54567b2c8-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.519488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.519498    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4476629-9b97-427c-82df-d7b54567b2c8-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:36.941320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.941271    4993 generic.go:356] "Generic (PLEG): container finished" podID="e4476629-9b97-427c-82df-d7b54567b2c8" containerID="e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019" exitCode=0
Oct 30 13:54:36.941744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.941355    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"
Oct 30 13:54:36.941744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.941358    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" event={"ID":"e4476629-9b97-427c-82df-d7b54567b2c8","Type":"ContainerDied","Data":"e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019"}
Oct 30 13:54:36.941744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.941491    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw" event={"ID":"e4476629-9b97-427c-82df-d7b54567b2c8","Type":"ContainerDied","Data":"f16e8cb79ce066578e534bab93b9f9b734d06cedaf57eeb06408b9230d735772"}
Oct 30 13:54:36.941744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.941523    4993 scope.go:122] "RemoveContainer" containerID="e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019"
Oct 30 13:54:36.956774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.956744    4993 scope.go:122] "RemoveContainer" containerID="c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6"
Oct 30 13:54:36.968861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.968838    4993 scope.go:122] "RemoveContainer" containerID="e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019"
Oct 30 13:54:36.969193 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:54:36.969172    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019\": container with ID starting with e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019 not found: ID does not exist" containerID="e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019"
Oct 30 13:54:36.969276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.969238    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019"} err="failed to get container status \"e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019\": rpc error: code = NotFound desc = could not find container \"e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019\": container with ID starting with e5b78c4b35723ad98d28ef434f3165b862edbb173031920d3551ca1c0162c019 not found: ID does not exist"
Oct 30 13:54:36.969276 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.969265    4993 scope.go:122] "RemoveContainer" containerID="c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6"
Oct 30 13:54:36.969570 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:54:36.969549    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6\": container with ID starting with c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6 not found: ID does not exist" containerID="c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6"
Oct 30 13:54:36.969653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:36.969577    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6"} err="failed to get container status \"c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6\": rpc error: code = NotFound desc = could not find container \"c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6\": container with ID starting with c5eea30692c24242a2b09988d233b2e9a3155b68a28bcec87f22b313e75d66f6 not found: ID does not exist"
Oct 30 13:54:37.009998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:37.009960    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"]
Oct 30 13:54:37.021119 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:37.021084    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-dnjrw"]
Oct 30 13:54:37.692077 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:37.692039    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" path="/var/lib/kubelet/pods/e4476629-9b97-427c-82df-d7b54567b2c8/volumes"
Oct 30 13:54:38.570396 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:54:38.570355    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:54:41.921695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.921655    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"]
Oct 30 13:54:41.922180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922166    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy"
Oct 30 13:54:41.922253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922182    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy"
Oct 30 13:54:41.922253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922192    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy"
Oct 30 13:54:41.922253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922198    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy"
Oct 30 13:54:41.922253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922250    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9d1ac26e-e2e8-4701-8fff-6d7dbca89df3" containerName="revoke-keys"
Oct 30 13:54:41.922378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922256    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1ac26e-e2e8-4701-8fff-6d7dbca89df3" containerName="revoke-keys"
Oct 30 13:54:41.922378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922264    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy"
Oct 30 13:54:41.922378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922269    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy"
Oct 30 13:54:41.922378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922276    4993 cpu_manager.go:414] "containerMap: removing container" podUID="674223a1-9a7f-4ea8-a6de-2f5850d3a14a" containerName="revoke-keys"
Oct 30 13:54:41.922378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922280    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674223a1-9a7f-4ea8-a6de-2f5850d3a14a" containerName="revoke-keys"
Oct 30 13:54:41.922378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922361    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy"
Oct 30 13:54:41.922378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922373    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1ac26e-e2e8-4701-8fff-6d7dbca89df3" containerName="revoke-keys"
Oct 30 13:54:41.922588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922381    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy"
Oct 30 13:54:41.922588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.922387    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="674223a1-9a7f-4ea8-a6de-2f5850d3a14a" containerName="revoke-keys"
Oct 30 13:54:41.927248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.927217    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:41.943639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.943608    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:54:41.949037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.949007    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"]
Oct 30 13:54:41.956263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.955775    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:54:41.956263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.955929    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:54:41.976577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.976541    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:54:41.996413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.995149    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9ng\" (UniqueName: \"kubernetes.io/projected/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-kube-api-access-xd9ng\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:41.996413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.995512    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:41.996413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:41.995581    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:42.096911 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:42.096868    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:42.097118 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:42.097045    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:42.097174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:42.097137    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9ng\" (UniqueName: \"kubernetes.io/projected/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-kube-api-access-xd9ng\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:42.097794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:42.097760    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:42.099706 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:42.099680    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:42.121262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:42.121229    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9ng\" (UniqueName: \"kubernetes.io/projected/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-kube-api-access-xd9ng\") pod \"maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:42.241278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:42.241243    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:42.556778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:42.556723    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"]
Oct 30 13:54:42.557127 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:54:42.557097    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201bb89e_17c0_40da_9b41_9ee4a1b8c5e4.slice/crio-f167191a8168a00e3187dc609a164a1408757e42627cba06c10a4810835f6860 WatchSource:0}: Error finding container f167191a8168a00e3187dc609a164a1408757e42627cba06c10a4810835f6860: Status 404 returned error can't find the container with id f167191a8168a00e3187dc609a164a1408757e42627cba06c10a4810835f6860
Oct 30 13:54:43.038162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:43.036809    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9" event={"ID":"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4","Type":"ContainerStarted","Data":"e169494395b6e0ea47a2ce386e68a29db1251d97341485f6fc596c4d4068a25d"}
Oct 30 13:54:43.038162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:43.036884    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9" event={"ID":"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4","Type":"ContainerStarted","Data":"f167191a8168a00e3187dc609a164a1408757e42627cba06c10a4810835f6860"}
Oct 30 13:54:43.079306 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:43.079198    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9" podStartSLOduration=2.079179413 podStartE2EDuration="2.079179413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:43.077512551 +0000 UTC m=+4947.553232161" watchObservedRunningTime="2026-10-30 13:54:43.079179413 +0000 UTC m=+4947.554899025"
Oct 30 13:54:44.046712 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:44.046667    4993 generic.go:356] "Generic (PLEG): container finished" podID="201bb89e-17c0-40da-9b41-9ee4a1b8c5e4" containerID="e169494395b6e0ea47a2ce386e68a29db1251d97341485f6fc596c4d4068a25d" exitCode=0
Oct 30 13:54:44.047177 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:44.046756    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9" event={"ID":"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4","Type":"ContainerDied","Data":"e169494395b6e0ea47a2ce386e68a29db1251d97341485f6fc596c4d4068a25d"}
Oct 30 13:54:45.240830 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.240797    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:45.355000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.354897    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-xks-ca\") pod \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") "
Oct 30 13:54:45.355175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.355050    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-openshift-ca\") pod \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") "
Oct 30 13:54:45.355175 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.355142    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-kube-api-access-xd9ng\" (UniqueName: \"kubernetes.io/projected/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-kube-api-access-xd9ng\") pod \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\" (UID: \"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4\") "
Oct 30 13:54:45.355496 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.355469    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-openshift-ca" pod "201bb89e-17c0-40da-9b41-9ee4a1b8c5e4" (UID: "201bb89e-17c0-40da-9b41-9ee4a1b8c5e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:45.355723 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.355704    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:45.357737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.357702    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-xks-ca" pod "201bb89e-17c0-40da-9b41-9ee4a1b8c5e4" (UID: "201bb89e-17c0-40da-9b41-9ee4a1b8c5e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:45.357829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.357753    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-kube-api-access-xd9ng" pod "201bb89e-17c0-40da-9b41-9ee4a1b8c5e4" (UID: "201bb89e-17c0-40da-9b41-9ee4a1b8c5e4"). InnerVolumeSpecName "kube-api-access-xd9ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:45.456729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.456683    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd9ng\" (UniqueName: \"kubernetes.io/projected/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-kube-api-access-xd9ng\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:45.456729 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:45.456723    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:46.066103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:46.066066    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9" event={"ID":"201bb89e-17c0-40da-9b41-9ee4a1b8c5e4","Type":"ContainerDied","Data":"f167191a8168a00e3187dc609a164a1408757e42627cba06c10a4810835f6860"}
Oct 30 13:54:46.066400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:46.066127    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f167191a8168a00e3187dc609a164a1408757e42627cba06c10a4810835f6860"
Oct 30 13:54:46.066400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:46.066074    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"
Oct 30 13:54:47.606284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:47.606241    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"]
Oct 30 13:54:47.606895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:47.606528    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy" containerID="cri-o://8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e" gracePeriod=30
Oct 30 13:54:47.615631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:47.615592    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"]
Oct 30 13:54:47.619195 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:47.619141    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy" containerID="cri-o://7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b" gracePeriod=30
Oct 30 13:54:52.951955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:52.951912    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:54:52.985081 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:52.985045    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069281    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-socket\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069366    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-token\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069432    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-podinfo\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069781    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-kube-api-access-v6zsm\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-kube-api-access-v6zsm\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069826    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-certs\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069824    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-socket" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069896    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-token\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-token\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069944    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-data\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.069974    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istiod-ca-cert\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070001    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-envoy\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070025    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d892df7c-4301-4f22-aaf8-ceca873df61b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d892df7c-4301-4f22-aaf8-ceca873df61b-istiod-ca-cert\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070054    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-data\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070107    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-socket\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070137    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-podinfo\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070162    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-credential-socket\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070272    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-credential-socket\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070303    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-kube-api-access-wbzsr\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-kube-api-access-wbzsr\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070332    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-certs\") pod \"d892df7c-4301-4f22-aaf8-ceca873df61b\" (UID: \"d892df7c-4301-4f22-aaf8-ceca873df61b\") "
Oct 30 13:54:53.070908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070367    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-envoy\") pod \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\" (UID: \"e2a75412-1fd2-4c36-a3b3-4d87736d48a4\") "
Oct 30 13:54:53.070908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.070886    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.071352 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.071101    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-certs" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.072488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.071699    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-data" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.072488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.072168    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d892df7c-4301-4f22-aaf8-ceca873df61b-istiod-ca-cert" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:53.072488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.072480    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-credential-socket" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.072750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.072718    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-socket" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.073914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.073883    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-credential-socket" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.075263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.074582    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-podinfo" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:54:53.075263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.074924    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-data" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.075263 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.075157    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-certs" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.076401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.076377    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istiod-ca-cert" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:53.076513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.076389    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-kube-api-access-wbzsr" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "kube-api-access-wbzsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:53.076818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.076796    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-envoy" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.076967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.076941    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-kube-api-access-v6zsm" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "kube-api-access-v6zsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:53.077093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.077070    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-token" pod "e2a75412-1fd2-4c36-a3b3-4d87736d48a4" (UID: "e2a75412-1fd2-4c36-a3b3-4d87736d48a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:53.078507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.078472    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-envoy" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:53.078675 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.078658    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-token" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:53.079394 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.079367    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-podinfo" pod "d892df7c-4301-4f22-aaf8-ceca873df61b" (UID: "d892df7c-4301-4f22-aaf8-ceca873df61b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:54:53.125528 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.125486    4993 generic.go:356] "Generic (PLEG): container finished" podID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerID="8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e" exitCode=0
Oct 30 13:54:53.125757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.125549    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"
Oct 30 13:54:53.125757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.125576    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" event={"ID":"d892df7c-4301-4f22-aaf8-ceca873df61b","Type":"ContainerDied","Data":"8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e"}
Oct 30 13:54:53.125757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.125667    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm" event={"ID":"d892df7c-4301-4f22-aaf8-ceca873df61b","Type":"ContainerDied","Data":"8c2c7261ded2448959474665a4735b3dc210510862023d7585cef8b954f27d21"}
Oct 30 13:54:53.125757 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.125699    4993 scope.go:122] "RemoveContainer" containerID="8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e"
Oct 30 13:54:53.128531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.128498    4993 generic.go:356] "Generic (PLEG): container finished" podID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerID="7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b" exitCode=0
Oct 30 13:54:53.128711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.128552    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"
Oct 30 13:54:53.128711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.128567    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" event={"ID":"e2a75412-1fd2-4c36-a3b3-4d87736d48a4","Type":"ContainerDied","Data":"7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b"}
Oct 30 13:54:53.128711 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.128626    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm" event={"ID":"e2a75412-1fd2-4c36-a3b3-4d87736d48a4","Type":"ContainerDied","Data":"c1029597e3e9575287135e23733b7110fce1f66231894c60203e3f15b38c2413"}
Oct 30 13:54:53.141466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.141439    4993 scope.go:122] "RemoveContainer" containerID="8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e"
Oct 30 13:54:53.141949 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:54:53.141834    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e\": container with ID starting with 8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e not found: ID does not exist" containerID="8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e"
Oct 30 13:54:53.141949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.141876    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e"} err="failed to get container status \"8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e\": rpc error: code = NotFound desc = could not find container \"8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e\": container with ID starting with 8b6c0c68316fe2622db189413ad50cb6cfd4f652f02b298817b570a2f616fc4e not found: ID does not exist"
Oct 30 13:54:53.141949 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.141902    4993 scope.go:122] "RemoveContainer" containerID="7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b"
Oct 30 13:54:53.154037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.154011    4993 scope.go:122] "RemoveContainer" containerID="7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b"
Oct 30 13:54:53.154398 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:54:53.154376    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b\": container with ID starting with 7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b not found: ID does not exist" containerID="7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b"
Oct 30 13:54:53.154473 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.154406    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b"} err="failed to get container status \"7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b\": rpc error: code = NotFound desc = could not find container \"7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b\": container with ID starting with 7ca552c427bbdf172efba06c64defc65abead6b521dd1bf323382ba56fef911b not found: ID does not exist"
Oct 30 13:54:53.172981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172875    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.172981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172911    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.172981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172923    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.172981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172934    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.172981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172947    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d892df7c-4301-4f22-aaf8-ceca873df61b-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.172981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172958    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.172981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172970    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.172981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172982    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d892df7c-4301-4f22-aaf8-ceca873df61b-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.172997    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.173010    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.173023    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbzsr\" (UniqueName: \"kubernetes.io/projected/d892df7c-4301-4f22-aaf8-ceca873df61b-kube-api-access-wbzsr\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.173034    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d892df7c-4301-4f22-aaf8-ceca873df61b-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.173045    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.173055    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.173068    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.173079    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6zsm\" (UniqueName: \"kubernetes.io/projected/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-kube-api-access-v6zsm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.174365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.173090    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75412-1fd2-4c36-a3b3-4d87736d48a4-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:54:53.246555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.246439    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"]
Oct 30 13:54:53.278344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.278307    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rfmxm"]
Oct 30 13:54:53.336595 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.336549    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"]
Oct 30 13:54:53.336762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.336620    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-f8kbm"]
Oct 30 13:54:53.692632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.692597    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" path="/var/lib/kubelet/pods/d892df7c-4301-4f22-aaf8-ceca873df61b/volumes"
Oct 30 13:54:53.693120 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:53.693105    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" path="/var/lib/kubelet/pods/e2a75412-1fd2-4c36-a3b3-4d87736d48a4/volumes"
Oct 30 13:54:59.593466 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:59.593416    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"]
Oct 30 13:54:59.602752 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:59.602711    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8740d4-xndmx"]
Oct 30 13:54:59.706915 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:54:59.706873    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674223a1-9a7f-4ea8-a6de-2f5850d3a14a" path="/var/lib/kubelet/pods/674223a1-9a7f-4ea8-a6de-2f5850d3a14a/volumes"
Oct 30 13:55:01.928148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:01.928104    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"]
Oct 30 13:55:01.939697 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:01.939653    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a8b38-czwt8"]
Oct 30 13:55:03.763449 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:03.763186    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1ac26e-e2e8-4701-8fff-6d7dbca89df3" path="/var/lib/kubelet/pods/9d1ac26e-e2e8-4701-8fff-6d7dbca89df3/volumes"
Oct 30 13:55:04.083086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:04.082984    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"]
Oct 30 13:55:04.107982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:04.107932    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1957dc-bwln9"]
Oct 30 13:55:05.692068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:05.692029    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201bb89e-17c0-40da-9b41-9ee4a1b8c5e4" path="/var/lib/kubelet/pods/201bb89e-17c0-40da-9b41-9ee4a1b8c5e4/volumes"
Oct 30 13:55:08.583290 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:55:08.583242    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:55:09.479307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479265    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"]
Oct 30 13:55:09.479790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479774    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy"
Oct 30 13:55:09.479840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479790    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy"
Oct 30 13:55:09.479840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479806    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy"
Oct 30 13:55:09.479840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479811    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy"
Oct 30 13:55:09.479939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479866    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy"
Oct 30 13:55:09.479939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479872    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy"
Oct 30 13:55:09.479939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479879    4993 cpu_manager.go:414] "containerMap: removing container" podUID="201bb89e-17c0-40da-9b41-9ee4a1b8c5e4" containerName="revoke-keys"
Oct 30 13:55:09.479939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479884    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201bb89e-17c0-40da-9b41-9ee4a1b8c5e4" containerName="revoke-keys"
Oct 30 13:55:09.480066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479960    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="201bb89e-17c0-40da-9b41-9ee4a1b8c5e4" containerName="revoke-keys"
Oct 30 13:55:09.480066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479973    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d892df7c-4301-4f22-aaf8-ceca873df61b" containerName="istio-proxy"
Oct 30 13:55:09.480066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479984    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a75412-1fd2-4c36-a3b3-4d87736d48a4" containerName="istio-proxy"
Oct 30 13:55:09.480066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.479995    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4476629-9b97-427c-82df-d7b54567b2c8" containerName="istio-proxy"
Oct 30 13:55:09.480066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.480012    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e3b0a0-6454-4b4e-bc88-9d4869ea0ed9" containerName="istio-proxy"
Oct 30 13:55:09.485456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.485421    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.515235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.514886    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"]
Oct 30 13:55:09.517914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.517589    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:55:09.518128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.518100    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:55:09.530265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.529108    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.530265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.529272    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.530265 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.529345    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6qp\" (UniqueName: \"kubernetes.io/projected/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-kube-api-access-4v6qp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.539587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.539543    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:55:09.539804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.539754    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:55:09.630719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.630674    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.631266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.630745    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6qp\" (UniqueName: \"kubernetes.io/projected/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-kube-api-access-4v6qp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.631266 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.630828    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.632024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.631604    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.633330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.633302    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.654168 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.654128    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6qp\" (UniqueName: \"kubernetes.io/projected/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-kube-api-access-4v6qp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:09.802998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:09.802889    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:10.114544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.114506    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"]
Oct 30 13:55:10.116058 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:55:10.116025    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00fb3a7_7fd1_4e08_80d2_cd8d71ea9360.slice/crio-c86a5f4da79ceb496d61588adae2319deda50be0eb29962e7dfe4eb3935546de WatchSource:0}: Error finding container c86a5f4da79ceb496d61588adae2319deda50be0eb29962e7dfe4eb3935546de: Status 404 returned error can't find the container with id c86a5f4da79ceb496d61588adae2319deda50be0eb29962e7dfe4eb3935546de
Oct 30 13:55:10.325667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.325626    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp" event={"ID":"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360","Type":"ContainerStarted","Data":"c86a5f4da79ceb496d61588adae2319deda50be0eb29962e7dfe4eb3935546de"}
Oct 30 13:55:10.522385 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.522341    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"]
Oct 30 13:55:10.526477 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.526452    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.573947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.573909    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"]
Oct 30 13:55:10.645531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.645443    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.645918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.645585    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.645918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.645664    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8x9\" (UniqueName: \"kubernetes.io/projected/80acf968-3c72-4f2a-bb7d-afb7bef1d820-kube-api-access-gc8x9\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.746933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.746889    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.747174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.746944    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8x9\" (UniqueName: \"kubernetes.io/projected/80acf968-3c72-4f2a-bb7d-afb7bef1d820-kube-api-access-gc8x9\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.747174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.747102    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.747831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.747801    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.749839 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.749810    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.770689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.770650    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8x9\" (UniqueName: \"kubernetes.io/projected/80acf968-3c72-4f2a-bb7d-afb7bef1d820-kube-api-access-gc8x9\") pod \"maas-api-revoke-sub-4ec788d2-tcv7x\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:10.870259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:10.865246    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:11.195284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.195247    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"]
Oct 30 13:55:11.234510 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:55:11.234473    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80acf968_3c72_4f2a_bb7d_afb7bef1d820.slice/crio-939484ebe3e9ae4d46fa42517f8c530080116a3ded9a3db04ee2ec526c4692fe WatchSource:0}: Error finding container 939484ebe3e9ae4d46fa42517f8c530080116a3ded9a3db04ee2ec526c4692fe: Status 404 returned error can't find the container with id 939484ebe3e9ae4d46fa42517f8c530080116a3ded9a3db04ee2ec526c4692fe
Oct 30 13:55:11.335463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.335409    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x" event={"ID":"80acf968-3c72-4f2a-bb7d-afb7bef1d820","Type":"ContainerStarted","Data":"939484ebe3e9ae4d46fa42517f8c530080116a3ded9a3db04ee2ec526c4692fe"}
Oct 30 13:55:11.337227 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.337183    4993 generic.go:356] "Generic (PLEG): container finished" podID="a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360" containerID="c27350d776037e8d300ddea5bc1e73b93b8019a511b39771d3e6219b0d10dc9c" exitCode=0
Oct 30 13:55:11.337376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.337255    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp" event={"ID":"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360","Type":"ContainerDied","Data":"c27350d776037e8d300ddea5bc1e73b93b8019a511b39771d3e6219b0d10dc9c"}
Oct 30 13:55:11.472158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.471780    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"]
Oct 30 13:55:11.478795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.478762    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.500005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.499959    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"]
Oct 30 13:55:11.583926 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.583888    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbhxc\" (UniqueName: \"kubernetes.io/projected/bbaf0b2a-6143-4f75-b86c-392a54575fa3-kube-api-access-pbhxc\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.584123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.584035    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.584123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.584103    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.685339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.685287    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbhxc\" (UniqueName: \"kubernetes.io/projected/bbaf0b2a-6143-4f75-b86c-392a54575fa3-kube-api-access-pbhxc\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.685797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.685407    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.685797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.685477    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.686376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.686350    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.688094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.688065    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.708983 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.708944    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbhxc\" (UniqueName: \"kubernetes.io/projected/bbaf0b2a-6143-4f75-b86c-392a54575fa3-kube-api-access-pbhxc\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:11.795391 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:11.795344    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:12.106399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.106368    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"]
Oct 30 13:55:12.108457 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:55:12.108423    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaf0b2a_6143_4f75_b86c_392a54575fa3.slice/crio-27ac4f3b3800a0fafeb26028d4f20051f4ae5b25ec66acd72b2494b4e17baccb WatchSource:0}: Error finding container 27ac4f3b3800a0fafeb26028d4f20051f4ae5b25ec66acd72b2494b4e17baccb: Status 404 returned error can't find the container with id 27ac4f3b3800a0fafeb26028d4f20051f4ae5b25ec66acd72b2494b4e17baccb
Oct 30 13:55:12.346868 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.346828    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4" event={"ID":"bbaf0b2a-6143-4f75-b86c-392a54575fa3","Type":"ContainerStarted","Data":"27ac4f3b3800a0fafeb26028d4f20051f4ae5b25ec66acd72b2494b4e17baccb"}
Oct 30 13:55:12.348727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.348697    4993 generic.go:356] "Generic (PLEG): container finished" podID="80acf968-3c72-4f2a-bb7d-afb7bef1d820" containerID="ea2cf3e4498ba5d7186a848d890c533d25f9f45ae95eb98c0d5be3926d6757ed" exitCode=0
Oct 30 13:55:12.348878 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.348777    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x" event={"ID":"80acf968-3c72-4f2a-bb7d-afb7bef1d820","Type":"ContainerDied","Data":"ea2cf3e4498ba5d7186a848d890c533d25f9f45ae95eb98c0d5be3926d6757ed"}
Oct 30 13:55:12.496814 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.496782    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:12.595668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.595635    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-kube-api-access-4v6qp\" (UniqueName: \"kubernetes.io/projected/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-kube-api-access-4v6qp\") pod \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") "
Oct 30 13:55:12.595832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.595743    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-openshift-ca\") pod \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") "
Oct 30 13:55:12.595929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.595912    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-xks-ca\") pod \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\" (UID: \"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360\") "
Oct 30 13:55:12.596190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.596159    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-openshift-ca" pod "a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360" (UID: "a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:12.596517 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.596490    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:12.598103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.598072    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-xks-ca" pod "a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360" (UID: "a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:12.598196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.598155    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-kube-api-access-4v6qp" pod "a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360" (UID: "a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360"). InnerVolumeSpecName "kube-api-access-4v6qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:12.697530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.697486    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6qp\" (UniqueName: \"kubernetes.io/projected/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-kube-api-access-4v6qp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:12.697530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:12.697526    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:13.211964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.211272    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"]
Oct 30 13:55:13.211964 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.211535    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy" containerID="cri-o://42338d650c6517db65f832080098669a437a373aae00d249974ad1373aba1d2c" gracePeriod=30
Oct 30 13:55:13.393295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.389587    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp" event={"ID":"a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360","Type":"ContainerDied","Data":"c86a5f4da79ceb496d61588adae2319deda50be0eb29962e7dfe4eb3935546de"}
Oct 30 13:55:13.393295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.389668    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86a5f4da79ceb496d61588adae2319deda50be0eb29962e7dfe4eb3935546de"
Oct 30 13:55:13.393295 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.390249    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"
Oct 30 13:55:13.402232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.393651    4993 generic.go:356] "Generic (PLEG): container finished" podID="bbaf0b2a-6143-4f75-b86c-392a54575fa3" containerID="82e344d868b54c56013db5fb02e5fac89e23823d7e07b02b5c10d6e3fb6e2bc1" exitCode=0
Oct 30 13:55:13.402232 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.395180    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4" event={"ID":"bbaf0b2a-6143-4f75-b86c-392a54575fa3","Type":"ContainerDied","Data":"82e344d868b54c56013db5fb02e5fac89e23823d7e07b02b5c10d6e3fb6e2bc1"}
Oct 30 13:55:13.592310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.592278    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:13.740348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.740307    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-xks-ca\") pod \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") "
Oct 30 13:55:13.740855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.740373    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80acf968-3c72-4f2a-bb7d-afb7bef1d820-kube-api-access-gc8x9\" (UniqueName: \"kubernetes.io/projected/80acf968-3c72-4f2a-bb7d-afb7bef1d820-kube-api-access-gc8x9\") pod \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") "
Oct 30 13:55:13.740855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.740587    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-openshift-ca\") pod \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\" (UID: \"80acf968-3c72-4f2a-bb7d-afb7bef1d820\") "
Oct 30 13:55:13.741052 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.740990    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-openshift-ca" pod "80acf968-3c72-4f2a-bb7d-afb7bef1d820" (UID: "80acf968-3c72-4f2a-bb7d-afb7bef1d820"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:13.742718 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.742689    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-xks-ca" pod "80acf968-3c72-4f2a-bb7d-afb7bef1d820" (UID: "80acf968-3c72-4f2a-bb7d-afb7bef1d820"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:13.742923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.742906    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80acf968-3c72-4f2a-bb7d-afb7bef1d820-kube-api-access-gc8x9" pod "80acf968-3c72-4f2a-bb7d-afb7bef1d820" (UID: "80acf968-3c72-4f2a-bb7d-afb7bef1d820"). InnerVolumeSpecName "kube-api-access-gc8x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:13.801488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.801446    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"]
Oct 30 13:55:13.802085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.802061    4993 cpu_manager.go:414] "containerMap: removing container" podUID="80acf968-3c72-4f2a-bb7d-afb7bef1d820" containerName="revoke-keys"
Oct 30 13:55:13.802228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.802087    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80acf968-3c72-4f2a-bb7d-afb7bef1d820" containerName="revoke-keys"
Oct 30 13:55:13.802228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.802122    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360" containerName="revoke-keys"
Oct 30 13:55:13.802228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.802131    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360" containerName="revoke-keys"
Oct 30 13:55:13.802413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.802314    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="80acf968-3c72-4f2a-bb7d-afb7bef1d820" containerName="revoke-keys"
Oct 30 13:55:13.802413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.802334    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360" containerName="revoke-keys"
Oct 30 13:55:13.805939 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.805907    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:13.829773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.829732    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"]
Oct 30 13:55:13.842601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.842562    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:13.842601 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.842596    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80acf968-3c72-4f2a-bb7d-afb7bef1d820-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:13.842880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.842610    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc8x9\" (UniqueName: \"kubernetes.io/projected/80acf968-3c72-4f2a-bb7d-afb7bef1d820-kube-api-access-gc8x9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:13.943396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.943340    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:13.943574 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.943482    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5kv\" (UniqueName: \"kubernetes.io/projected/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-kube-api-access-zt5kv\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:13.943659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:13.943643    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:14.044818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.044711    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:14.044975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.044861    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:14.044975 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.044953    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5kv\" (UniqueName: \"kubernetes.io/projected/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-kube-api-access-zt5kv\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:14.045761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.045726    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:14.047637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.047605    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:14.069250 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.069165    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5kv\" (UniqueName: \"kubernetes.io/projected/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-kube-api-access-zt5kv\") pod \"maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:14.121392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.121350    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:14.403514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.403426    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x"
Oct 30 13:55:14.403514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.403444    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec788d2-tcv7x" event={"ID":"80acf968-3c72-4f2a-bb7d-afb7bef1d820","Type":"ContainerDied","Data":"939484ebe3e9ae4d46fa42517f8c530080116a3ded9a3db04ee2ec526c4692fe"}
Oct 30 13:55:14.403514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.403491    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="939484ebe3e9ae4d46fa42517f8c530080116a3ded9a3db04ee2ec526c4692fe"
Oct 30 13:55:14.438967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.438931    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"]
Oct 30 13:55:14.440837 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:55:14.440807    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1b65a9_4845_4ae7_a4a3_5c0f2b581f8f.slice/crio-27d5b76fb9eaf0b28ab73bd02ddb4ad89b89d5617adffab68a43d01000594efa WatchSource:0}: Error finding container 27d5b76fb9eaf0b28ab73bd02ddb4ad89b89d5617adffab68a43d01000594efa: Status 404 returned error can't find the container with id 27d5b76fb9eaf0b28ab73bd02ddb4ad89b89d5617adffab68a43d01000594efa
Oct 30 13:55:14.558110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.558075    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:14.651767 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.651727    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-openshift-ca\") pod \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") "
Oct 30 13:55:14.651938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.651803    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbaf0b2a-6143-4f75-b86c-392a54575fa3-kube-api-access-pbhxc\" (UniqueName: \"kubernetes.io/projected/bbaf0b2a-6143-4f75-b86c-392a54575fa3-kube-api-access-pbhxc\") pod \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") "
Oct 30 13:55:14.651938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.651829    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-xks-ca\") pod \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\" (UID: \"bbaf0b2a-6143-4f75-b86c-392a54575fa3\") "
Oct 30 13:55:14.652123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.652099    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-openshift-ca" pod "bbaf0b2a-6143-4f75-b86c-392a54575fa3" (UID: "bbaf0b2a-6143-4f75-b86c-392a54575fa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:14.652480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.652452    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:14.654192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.654126    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-xks-ca" pod "bbaf0b2a-6143-4f75-b86c-392a54575fa3" (UID: "bbaf0b2a-6143-4f75-b86c-392a54575fa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:14.654495 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.654471    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbaf0b2a-6143-4f75-b86c-392a54575fa3-kube-api-access-pbhxc" pod "bbaf0b2a-6143-4f75-b86c-392a54575fa3" (UID: "bbaf0b2a-6143-4f75-b86c-392a54575fa3"). InnerVolumeSpecName "kube-api-access-pbhxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:14.754504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.754085    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbhxc\" (UniqueName: \"kubernetes.io/projected/bbaf0b2a-6143-4f75-b86c-392a54575fa3-kube-api-access-pbhxc\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:14.754504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:14.754125    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf0b2a-6143-4f75-b86c-392a54575fa3-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:15.412288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:15.412251    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"
Oct 30 13:55:15.412288 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:15.412265    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4" event={"ID":"bbaf0b2a-6143-4f75-b86c-392a54575fa3","Type":"ContainerDied","Data":"27ac4f3b3800a0fafeb26028d4f20051f4ae5b25ec66acd72b2494b4e17baccb"}
Oct 30 13:55:15.413157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:15.412331    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ac4f3b3800a0fafeb26028d4f20051f4ae5b25ec66acd72b2494b4e17baccb"
Oct 30 13:55:15.414253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:15.414181    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h" event={"ID":"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f","Type":"ContainerStarted","Data":"e56d3e7a626f16d6ae26a6ab57b1477a503aba6511a39c8ff5e0be22bf88f376"}
Oct 30 13:55:15.414253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:15.414246    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h" event={"ID":"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f","Type":"ContainerStarted","Data":"27d5b76fb9eaf0b28ab73bd02ddb4ad89b89d5617adffab68a43d01000594efa"}
Oct 30 13:55:15.458834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:15.458762    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h" podStartSLOduration=2.45874365 podStartE2EDuration="2.45874365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:15.455750372 +0000 UTC m=+4979.931469983" watchObservedRunningTime="2026-10-30 13:55:15.45874365 +0000 UTC m=+4979.934463261"
Oct 30 13:55:16.424138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:16.424099    4993 generic.go:356] "Generic (PLEG): container finished" podID="cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f" containerID="e56d3e7a626f16d6ae26a6ab57b1477a503aba6511a39c8ff5e0be22bf88f376" exitCode=0
Oct 30 13:55:16.424634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:16.424151    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h" event={"ID":"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f","Type":"ContainerDied","Data":"e56d3e7a626f16d6ae26a6ab57b1477a503aba6511a39c8ff5e0be22bf88f376"}
Oct 30 13:55:17.600315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.600280    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:17.703320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.703222    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-xks-ca\") pod \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") "
Oct 30 13:55:17.703514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.703401    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-kube-api-access-zt5kv\" (UniqueName: \"kubernetes.io/projected/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-kube-api-access-zt5kv\") pod \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") "
Oct 30 13:55:17.703514 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.703480    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-openshift-ca\") pod \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\" (UID: \"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f\") "
Oct 30 13:55:17.703866 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.703824    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-openshift-ca" pod "cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f" (UID: "cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:17.704039 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.704019    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:17.705766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.705736    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-kube-api-access-zt5kv" pod "cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f" (UID: "cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f"). InnerVolumeSpecName "kube-api-access-zt5kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:17.705908 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.705890    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-xks-ca" pod "cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f" (UID: "cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:17.804877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.804842    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:17.804877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:17.804875    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt5kv\" (UniqueName: \"kubernetes.io/projected/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f-kube-api-access-zt5kv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.446836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.442723    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"
Oct 30 13:55:18.446836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.442740    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h" event={"ID":"cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f","Type":"ContainerDied","Data":"27d5b76fb9eaf0b28ab73bd02ddb4ad89b89d5617adffab68a43d01000594efa"}
Oct 30 13:55:18.446836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.442807    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d5b76fb9eaf0b28ab73bd02ddb4ad89b89d5617adffab68a43d01000594efa"
Oct 30 13:55:18.446836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.446032    4993 generic.go:356] "Generic (PLEG): container finished" podID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerID="42338d650c6517db65f832080098669a437a373aae00d249974ad1373aba1d2c" exitCode=0
Oct 30 13:55:18.446836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.446090    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" event={"ID":"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f","Type":"ContainerDied","Data":"42338d650c6517db65f832080098669a437a373aae00d249974ad1373aba1d2c"}
Oct 30 13:55:18.446836 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.446151    4993 scope.go:122] "RemoveContainer" containerID="445105b16cd4eefcd5e228ac2c8d77673656cfd8ad6447f65184bdde3aebcbc6"
Oct 30 13:55:18.500247 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.500215    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:55:18.615753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.615718    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-token\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-token\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.615766    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-podinfo\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.615806    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-data\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.615831    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istiod-ca-cert\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.615898    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-kube-api-access-wgqth\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-kube-api-access-wgqth\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.615931    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-envoy\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.616011    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-certs\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.616094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-socket\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616301 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.616228    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-data" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:18.616792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.616496    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-certs" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:18.616792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.616505    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-socket" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:18.616792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.616536    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-credential-socket\") pod \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\" (UID: \"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f\") "
Oct 30 13:55:18.616792 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.616756    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-credential-socket" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:18.617066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.617042    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.617066 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.617061    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.617196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.617071    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.617196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.617080    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.617458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.617431    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istiod-ca-cert" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:18.618631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.618595    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-podinfo" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:18.618996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.618963    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-envoy" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:18.619994 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.619961    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-kube-api-access-wgqth" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "kube-api-access-wgqth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:18.620577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.620552    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-token" pod "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" (UID: "d60c06df-f4cf-43c2-ba80-c100ea8c2b8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:18.718018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.717978    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.718018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.718014    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.718350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.718030    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.718350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.718042    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgqth\" (UniqueName: \"kubernetes.io/projected/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-kube-api-access-wgqth\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.718350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:18.718055    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:19.455670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:19.455636    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"
Oct 30 13:55:19.456015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:19.455676    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2" event={"ID":"d60c06df-f4cf-43c2-ba80-c100ea8c2b8f","Type":"ContainerDied","Data":"dc1da44ad0586a37b3659c80c1001e3fc1606519bffc73f8a2dbf983fb667ccd"}
Oct 30 13:55:19.456015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:19.455733    4993 scope.go:122] "RemoveContainer" containerID="42338d650c6517db65f832080098669a437a373aae00d249974ad1373aba1d2c"
Oct 30 13:55:19.521273 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:19.521237    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"]
Oct 30 13:55:19.532745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:19.532711    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1957dc-openshift-default-7d8748ffcc-bg9m2"]
Oct 30 13:55:19.692457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:19.692420    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" path="/var/lib/kubelet/pods/d60c06df-f4cf-43c2-ba80-c100ea8c2b8f/volumes"
Oct 30 13:55:20.850099 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850056    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"]
Oct 30 13:55:20.850587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850572    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bbaf0b2a-6143-4f75-b86c-392a54575fa3" containerName="revoke-keys"
Oct 30 13:55:20.850639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850588    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbaf0b2a-6143-4f75-b86c-392a54575fa3" containerName="revoke-keys"
Oct 30 13:55:20.850639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850610    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy"
Oct 30 13:55:20.850639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850616    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy"
Oct 30 13:55:20.850639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850631    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f" containerName="revoke-keys"
Oct 30 13:55:20.850639 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850637    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f" containerName="revoke-keys"
Oct 30 13:55:20.850809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850733    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbaf0b2a-6143-4f75-b86c-392a54575fa3" containerName="revoke-keys"
Oct 30 13:55:20.850809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850745    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy"
Oct 30 13:55:20.850809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850752    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f" containerName="revoke-keys"
Oct 30 13:55:20.850809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.850762    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy"
Oct 30 13:55:20.855822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.855791    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:20.883751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.883715    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:55:20.884000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.883849    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:55:20.884094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.884059    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:55:20.884160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.884110    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:55:20.890136 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.890093    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"]
Oct 30 13:55:20.949256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.949170    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c93a6d-c980-4497-8ce5-826281996589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:20.949487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.949368    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c93a6d-c980-4497-8ce5-826281996589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:20.949487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:20.949431    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzds5\" (UniqueName: \"kubernetes.io/projected/27c93a6d-c980-4497-8ce5-826281996589-kube-api-access-jzds5\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:21.050428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:21.050387    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c93a6d-c980-4497-8ce5-826281996589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:21.050641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:21.050443    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzds5\" (UniqueName: \"kubernetes.io/projected/27c93a6d-c980-4497-8ce5-826281996589-kube-api-access-jzds5\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:21.050641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:21.050582    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c93a6d-c980-4497-8ce5-826281996589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:21.051253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:21.051193    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c93a6d-c980-4497-8ce5-826281996589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:21.053011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:21.052987    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c93a6d-c980-4497-8ce5-826281996589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:21.074330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:21.074293    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzds5\" (UniqueName: \"kubernetes.io/projected/27c93a6d-c980-4497-8ce5-826281996589-kube-api-access-jzds5\") pod \"maas-api-revoke-sub-835a9ed8-b6bm9\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:21.173930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:21.173818    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:21.486481 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:21.486446    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"]
Oct 30 13:55:21.487946 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:55:21.487914    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c93a6d_c980_4497_8ce5_826281996589.slice/crio-b843a68b7d47134ac0593b79b64783d3b99a116aabe7c92f74af60cf99f01ed2 WatchSource:0}: Error finding container b843a68b7d47134ac0593b79b64783d3b99a116aabe7c92f74af60cf99f01ed2: Status 404 returned error can't find the container with id b843a68b7d47134ac0593b79b64783d3b99a116aabe7c92f74af60cf99f01ed2
Oct 30 13:55:22.507246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:22.505520    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9" event={"ID":"27c93a6d-c980-4497-8ce5-826281996589","Type":"ContainerStarted","Data":"44e2c1639d6218f424d3e523b4eafb1bc9933064f2fd62f9fd2a2f59bf3772f1"}
Oct 30 13:55:22.507246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:22.505631    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9" event={"ID":"27c93a6d-c980-4497-8ce5-826281996589","Type":"ContainerStarted","Data":"b843a68b7d47134ac0593b79b64783d3b99a116aabe7c92f74af60cf99f01ed2"}
Oct 30 13:55:22.549228 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:22.548685    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9" podStartSLOduration=2.548661697 podStartE2EDuration="2.548661697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:22.545437136 +0000 UTC m=+4987.021156746" watchObservedRunningTime="2026-10-30 13:55:22.548661697 +0000 UTC m=+4987.024381309"
Oct 30 13:55:23.516414 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:23.516374    4993 generic.go:356] "Generic (PLEG): container finished" podID="27c93a6d-c980-4497-8ce5-826281996589" containerID="44e2c1639d6218f424d3e523b4eafb1bc9933064f2fd62f9fd2a2f59bf3772f1" exitCode=0
Oct 30 13:55:23.516865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:23.516427    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9" event={"ID":"27c93a6d-c980-4497-8ce5-826281996589","Type":"ContainerDied","Data":"44e2c1639d6218f424d3e523b4eafb1bc9933064f2fd62f9fd2a2f59bf3772f1"}
Oct 30 13:55:24.183190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.183151    4993 scope.go:122] "RemoveContainer" containerID="0e3d3f84f12699f97d6d597418b54c57163ceb8b25f1239f1cae24c558b0c060"
Oct 30 13:55:24.708109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.708075    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:24.825852 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.825761    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c93a6d-c980-4497-8ce5-826281996589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c93a6d-c980-4497-8ce5-826281996589-maas-api-openshift-ca\") pod \"27c93a6d-c980-4497-8ce5-826281996589\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") "
Oct 30 13:55:24.826037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.825926    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c93a6d-c980-4497-8ce5-826281996589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c93a6d-c980-4497-8ce5-826281996589-maas-api-xks-ca\") pod \"27c93a6d-c980-4497-8ce5-826281996589\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") "
Oct 30 13:55:24.826098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.826049    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c93a6d-c980-4497-8ce5-826281996589-kube-api-access-jzds5\" (UniqueName: \"kubernetes.io/projected/27c93a6d-c980-4497-8ce5-826281996589-kube-api-access-jzds5\") pod \"27c93a6d-c980-4497-8ce5-826281996589\" (UID: \"27c93a6d-c980-4497-8ce5-826281996589\") "
Oct 30 13:55:24.826150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.826128    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c93a6d-c980-4497-8ce5-826281996589-maas-api-openshift-ca" pod "27c93a6d-c980-4497-8ce5-826281996589" (UID: "27c93a6d-c980-4497-8ce5-826281996589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:24.826633 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.826607    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c93a6d-c980-4497-8ce5-826281996589-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:24.828245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.828187    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c93a6d-c980-4497-8ce5-826281996589-kube-api-access-jzds5" pod "27c93a6d-c980-4497-8ce5-826281996589" (UID: "27c93a6d-c980-4497-8ce5-826281996589"). InnerVolumeSpecName "kube-api-access-jzds5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:24.828339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.828310    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c93a6d-c980-4497-8ce5-826281996589-maas-api-xks-ca" pod "27c93a6d-c980-4497-8ce5-826281996589" (UID: "27c93a6d-c980-4497-8ce5-826281996589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:24.928246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.928125    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c93a6d-c980-4497-8ce5-826281996589-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:24.928246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:24.928178    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzds5\" (UniqueName: \"kubernetes.io/projected/27c93a6d-c980-4497-8ce5-826281996589-kube-api-access-jzds5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:25.544461 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:25.544426    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9"
Oct 30 13:55:25.544658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:25.544457    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-835a9ed8-b6bm9" event={"ID":"27c93a6d-c980-4497-8ce5-826281996589","Type":"ContainerDied","Data":"b843a68b7d47134ac0593b79b64783d3b99a116aabe7c92f74af60cf99f01ed2"}
Oct 30 13:55:25.544658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:25.544511    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b843a68b7d47134ac0593b79b64783d3b99a116aabe7c92f74af60cf99f01ed2"
Oct 30 13:55:31.848160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:31.848114    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"]
Oct 30 13:55:33.861546 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:33.861500    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"]
Oct 30 13:55:33.873127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:33.873089    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-98dabb-nc8l4"]
Oct 30 13:55:35.692898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:35.692862    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbaf0b2a-6143-4f75-b86c-392a54575fa3" path="/var/lib/kubelet/pods/bbaf0b2a-6143-4f75-b86c-392a54575fa3/volumes"
Oct 30 13:55:38.576486 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:55:38.576442    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:55:38.956530 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.956440    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"]
Oct 30 13:55:38.956995 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.956979    4993 cpu_manager.go:414] "containerMap: removing container" podUID="27c93a6d-c980-4497-8ce5-826281996589" containerName="revoke-keys"
Oct 30 13:55:38.957044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.956997    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c93a6d-c980-4497-8ce5-826281996589" containerName="revoke-keys"
Oct 30 13:55:38.957044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.957038    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy"
Oct 30 13:55:38.957115 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.957043    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60c06df-f4cf-43c2-ba80-c100ea8c2b8f" containerName="istio-proxy"
Oct 30 13:55:38.957158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.957123    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c93a6d-c980-4497-8ce5-826281996589" containerName="revoke-keys"
Oct 30 13:55:38.960520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.960487    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:38.985634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.985594    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"]
Oct 30 13:55:38.989929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.989880    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:55:38.990142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.989969    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:55:38.990142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.990062    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:55:38.990142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:38.990135    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:55:39.035264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.035221    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.035264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.035265    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.035591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.035393    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k998\" (UniqueName: \"kubernetes.io/projected/2e830b61-6e31-4e88-aec3-3f69597a21e7-kube-api-access-9k998\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.137419 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.137365    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.137653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.137432    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.137653 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.137520    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k998\" (UniqueName: \"kubernetes.io/projected/2e830b61-6e31-4e88-aec3-3f69597a21e7-kube-api-access-9k998\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.138246 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.138193    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.140433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.140404    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.175049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.175016    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k998\" (UniqueName: \"kubernetes.io/projected/2e830b61-6e31-4e88-aec3-3f69597a21e7-kube-api-access-9k998\") pod \"maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.279006 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.278961    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:39.461278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.460631    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"]
Oct 30 13:55:39.466019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.465986    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2d2a0b-tvx7h"]
Oct 30 13:55:39.623234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.611762    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"]
Oct 30 13:55:39.734346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.734303    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f" path="/var/lib/kubelet/pods/cb1b65a9-4845-4ae7-a4a3-5c0f2b581f8f/volumes"
Oct 30 13:55:39.749518 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:39.749465    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g" event={"ID":"2e830b61-6e31-4e88-aec3-3f69597a21e7","Type":"ContainerStarted","Data":"5d6052580e490dfc91c3aef7da981755967188c9f6884ac0e46af39242b258d3"}
Oct 30 13:55:40.000736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:40.000686    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"]
Oct 30 13:55:40.006606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:40.006558    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e9cb4a-skxhp"]
Oct 30 13:55:40.762463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:40.762182    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g" event={"ID":"2e830b61-6e31-4e88-aec3-3f69597a21e7","Type":"ContainerStarted","Data":"112f70b9302b4f2a73adf7959713f8b711da6a437673fe42013090927739483d"}
Oct 30 13:55:40.805536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:40.805473    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g" podStartSLOduration=2.805454067 podStartE2EDuration="2.805454067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:40.802018856 +0000 UTC m=+5005.277738467" watchObservedRunningTime="2026-10-30 13:55:40.805454067 +0000 UTC m=+5005.281173716"
Oct 30 13:55:41.692122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:41.692087    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360" path="/var/lib/kubelet/pods/a00fb3a7-7fd1-4e08-80d2-cd8d71ea9360/volumes"
Oct 30 13:55:41.795432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:41.795387    4993 generic.go:356] "Generic (PLEG): container finished" podID="2e830b61-6e31-4e88-aec3-3f69597a21e7" containerID="112f70b9302b4f2a73adf7959713f8b711da6a437673fe42013090927739483d" exitCode=0
Oct 30 13:55:41.796031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:41.795624    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g" event={"ID":"2e830b61-6e31-4e88-aec3-3f69597a21e7","Type":"ContainerDied","Data":"112f70b9302b4f2a73adf7959713f8b711da6a437673fe42013090927739483d"}
Oct 30 13:55:42.559587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.559543    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"]
Oct 30 13:55:42.563973 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.563943    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.586294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.586256    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"]
Oct 30 13:55:42.594765 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.594718    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqgv\" (UniqueName: \"kubernetes.io/projected/642ac614-b975-403c-ad34-dfd6074b9692-kube-api-access-lqqgv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.594991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.594801    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.595456 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.595018    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.696153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.696104    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.696389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.696250    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqgv\" (UniqueName: \"kubernetes.io/projected/642ac614-b975-403c-ad34-dfd6074b9692-kube-api-access-lqqgv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.696389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.696312    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.696920 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.696891    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.699092 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.699063    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.721788 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.721755    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqgv\" (UniqueName: \"kubernetes.io/projected/642ac614-b975-403c-ad34-dfd6074b9692-kube-api-access-lqqgv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.886822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.884809    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:42.976641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.976188    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:43.000032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:42.999984    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-openshift-ca\") pod \"2e830b61-6e31-4e88-aec3-3f69597a21e7\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") "
Oct 30 13:55:43.000835 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.000712    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-openshift-ca" pod "2e830b61-6e31-4e88-aec3-3f69597a21e7" (UID: "2e830b61-6e31-4e88-aec3-3f69597a21e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:43.001079 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.001063    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-xks-ca\") pod \"2e830b61-6e31-4e88-aec3-3f69597a21e7\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") "
Oct 30 13:55:43.001589 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.001575    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e830b61-6e31-4e88-aec3-3f69597a21e7-kube-api-access-9k998\" (UniqueName: \"kubernetes.io/projected/2e830b61-6e31-4e88-aec3-3f69597a21e7-kube-api-access-9k998\") pod \"2e830b61-6e31-4e88-aec3-3f69597a21e7\" (UID: \"2e830b61-6e31-4e88-aec3-3f69597a21e7\") "
Oct 30 13:55:43.002236 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.002200    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:43.003400 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.003365    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-xks-ca" pod "2e830b61-6e31-4e88-aec3-3f69597a21e7" (UID: "2e830b61-6e31-4e88-aec3-3f69597a21e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:43.003903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.003880    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e830b61-6e31-4e88-aec3-3f69597a21e7-kube-api-access-9k998" pod "2e830b61-6e31-4e88-aec3-3f69597a21e7" (UID: "2e830b61-6e31-4e88-aec3-3f69597a21e7"). InnerVolumeSpecName "kube-api-access-9k998". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:43.104008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.103969    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e830b61-6e31-4e88-aec3-3f69597a21e7-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:43.104008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.104003    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k998\" (UniqueName: \"kubernetes.io/projected/2e830b61-6e31-4e88-aec3-3f69597a21e7-kube-api-access-9k998\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:43.212031 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.212001    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"]
Oct 30 13:55:43.214062 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:55:43.214034    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642ac614_b975_403c_ad34_dfd6074b9692.slice/crio-6d2ba891156895f488c7fd9cba5a7b07a17f9cb02a626b8f78ed50c789f261a7 WatchSource:0}: Error finding container 6d2ba891156895f488c7fd9cba5a7b07a17f9cb02a626b8f78ed50c789f261a7: Status 404 returned error can't find the container with id 6d2ba891156895f488c7fd9cba5a7b07a17f9cb02a626b8f78ed50c789f261a7
Oct 30 13:55:43.814996 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.814960    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"
Oct 30 13:55:43.815251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.814959    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g" event={"ID":"2e830b61-6e31-4e88-aec3-3f69597a21e7","Type":"ContainerDied","Data":"5d6052580e490dfc91c3aef7da981755967188c9f6884ac0e46af39242b258d3"}
Oct 30 13:55:43.815251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.815085    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d6052580e490dfc91c3aef7da981755967188c9f6884ac0e46af39242b258d3"
Oct 30 13:55:43.816954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.816922    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx" event={"ID":"642ac614-b975-403c-ad34-dfd6074b9692","Type":"ContainerStarted","Data":"6eeda28d79ec417bdb94f653910b87ffa9b60c604fc010cf8829adf140f18a55"}
Oct 30 13:55:43.817110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.816981    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx" event={"ID":"642ac614-b975-403c-ad34-dfd6074b9692","Type":"ContainerStarted","Data":"6d2ba891156895f488c7fd9cba5a7b07a17f9cb02a626b8f78ed50c789f261a7"}
Oct 30 13:55:43.862919 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:43.862836    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx" podStartSLOduration=1.8628136560000002 podStartE2EDuration="1.862813656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:43.857000175 +0000 UTC m=+5008.332719808" watchObservedRunningTime="2026-10-30 13:55:43.862813656 +0000 UTC m=+5008.338533269"
Oct 30 13:55:44.827014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:44.826974    4993 generic.go:356] "Generic (PLEG): container finished" podID="642ac614-b975-403c-ad34-dfd6074b9692" containerID="6eeda28d79ec417bdb94f653910b87ffa9b60c604fc010cf8829adf140f18a55" exitCode=0
Oct 30 13:55:44.827451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:44.827048    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx" event={"ID":"642ac614-b975-403c-ad34-dfd6074b9692","Type":"ContainerDied","Data":"6eeda28d79ec417bdb94f653910b87ffa9b60c604fc010cf8829adf140f18a55"}
Oct 30 13:55:46.077084 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.077040    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:46.143271 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.143128    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-openshift-ca\") pod \"642ac614-b975-403c-ad34-dfd6074b9692\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") "
Oct 30 13:55:46.143475 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.143367    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-xks-ca\") pod \"642ac614-b975-403c-ad34-dfd6074b9692\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") "
Oct 30 13:55:46.143562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.143495    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642ac614-b975-403c-ad34-dfd6074b9692-kube-api-access-lqqgv\" (UniqueName: \"kubernetes.io/projected/642ac614-b975-403c-ad34-dfd6074b9692-kube-api-access-lqqgv\") pod \"642ac614-b975-403c-ad34-dfd6074b9692\" (UID: \"642ac614-b975-403c-ad34-dfd6074b9692\") "
Oct 30 13:55:46.143703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.143676    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-openshift-ca" pod "642ac614-b975-403c-ad34-dfd6074b9692" (UID: "642ac614-b975-403c-ad34-dfd6074b9692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:46.144014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.143986    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.145925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.145894    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-xks-ca" pod "642ac614-b975-403c-ad34-dfd6074b9692" (UID: "642ac614-b975-403c-ad34-dfd6074b9692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:46.146049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.146024    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642ac614-b975-403c-ad34-dfd6074b9692-kube-api-access-lqqgv" pod "642ac614-b975-403c-ad34-dfd6074b9692" (UID: "642ac614-b975-403c-ad34-dfd6074b9692"). InnerVolumeSpecName "kube-api-access-lqqgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:46.245007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.244960    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642ac614-b975-403c-ad34-dfd6074b9692-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.245007 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.244998    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqqgv\" (UniqueName: \"kubernetes.io/projected/642ac614-b975-403c-ad34-dfd6074b9692-kube-api-access-lqqgv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.864346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.864315    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"
Oct 30 13:55:46.864566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.864358    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx" event={"ID":"642ac614-b975-403c-ad34-dfd6074b9692","Type":"ContainerDied","Data":"6d2ba891156895f488c7fd9cba5a7b07a17f9cb02a626b8f78ed50c789f261a7"}
Oct 30 13:55:46.864566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.864415    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d2ba891156895f488c7fd9cba5a7b07a17f9cb02a626b8f78ed50c789f261a7"
Oct 30 13:55:46.872370 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:46.872323    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4" podUID="6059e369-d2ed-4f31-8409-624fe832c897" containerName="main" containerID="cri-o://f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41" gracePeriod=45
Oct 30 13:55:47.218442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.218401    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:55:47.269599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.269454    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6059e369-d2ed-4f31-8409-624fe832c897-kube-api-access-xjvgt\" (UniqueName: \"kubernetes.io/projected/6059e369-d2ed-4f31-8409-624fe832c897-kube-api-access-xjvgt\") pod \"6059e369-d2ed-4f31-8409-624fe832c897\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") "
Oct 30 13:55:47.269945 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.269917    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6059e369-d2ed-4f31-8409-624fe832c897-tls-certs\" (UniqueName: \"kubernetes.io/secret/6059e369-d2ed-4f31-8409-624fe832c897-tls-certs\") pod \"6059e369-d2ed-4f31-8409-624fe832c897\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") "
Oct 30 13:55:47.270088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.270030    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-home\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-home\") pod \"6059e369-d2ed-4f31-8409-624fe832c897\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") "
Oct 30 13:55:47.270163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.270128    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-model-cache\") pod \"6059e369-d2ed-4f31-8409-624fe832c897\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") "
Oct 30 13:55:47.270299 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.270278    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-dshm\") pod \"6059e369-d2ed-4f31-8409-624fe832c897\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") "
Oct 30 13:55:47.270366 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.270357    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-tmp-dir\") pod \"6059e369-d2ed-4f31-8409-624fe832c897\" (UID: \"6059e369-d2ed-4f31-8409-624fe832c897\") "
Oct 30 13:55:47.271344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.271031    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-model-cache" pod "6059e369-d2ed-4f31-8409-624fe832c897" (UID: "6059e369-d2ed-4f31-8409-624fe832c897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.271344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.271048    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-tmp-dir" pod "6059e369-d2ed-4f31-8409-624fe832c897" (UID: "6059e369-d2ed-4f31-8409-624fe832c897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.271822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.271801    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-home" pod "6059e369-d2ed-4f31-8409-624fe832c897" (UID: "6059e369-d2ed-4f31-8409-624fe832c897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.272747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.272722    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6059e369-d2ed-4f31-8409-624fe832c897-kube-api-access-xjvgt" pod "6059e369-d2ed-4f31-8409-624fe832c897" (UID: "6059e369-d2ed-4f31-8409-624fe832c897"). InnerVolumeSpecName "kube-api-access-xjvgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:47.273277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.273249    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-dshm" pod "6059e369-d2ed-4f31-8409-624fe832c897" (UID: "6059e369-d2ed-4f31-8409-624fe832c897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.273605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.273571    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6059e369-d2ed-4f31-8409-624fe832c897-tls-certs" pod "6059e369-d2ed-4f31-8409-624fe832c897" (UID: "6059e369-d2ed-4f31-8409-624fe832c897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:47.315692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.315641    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"]
Oct 30 13:55:47.344914 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.344873    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"]
Oct 30 13:55:47.373310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.371782    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.373310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.371821    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.373310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.371835    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjvgt\" (UniqueName: \"kubernetes.io/projected/6059e369-d2ed-4f31-8409-624fe832c897-kube-api-access-xjvgt\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.373310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.371848    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6059e369-d2ed-4f31-8409-624fe832c897-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.373310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.371864    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.373310 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.371875    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6059e369-d2ed-4f31-8409-624fe832c897-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.875892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.875849    4993 generic.go:356] "Generic (PLEG): container finished" podID="6059e369-d2ed-4f31-8409-624fe832c897" containerID="f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41" exitCode=0
Oct 30 13:55:47.875892 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.875892    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"
Oct 30 13:55:47.876126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.875938    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4" event={"ID":"6059e369-d2ed-4f31-8409-624fe832c897","Type":"ContainerDied","Data":"f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41"}
Oct 30 13:55:47.876126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.875990    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4" event={"ID":"6059e369-d2ed-4f31-8409-624fe832c897","Type":"ContainerDied","Data":"b9b969a24ad974e557ce3adffd7925825f7c7e0e5e26a163bde05ad8377f2443"}
Oct 30 13:55:47.876126 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.876014    4993 scope.go:122] "RemoveContainer" containerID="f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41"
Oct 30 13:55:47.890880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.890832    4993 scope.go:122] "RemoveContainer" containerID="f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41"
Oct 30 13:55:47.891326 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:55:47.891292    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41\": container with ID starting with f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41 not found: ID does not exist" containerID="f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41"
Oct 30 13:55:47.891468 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.891334    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41"} err="failed to get container status \"f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41\": rpc error: code = NotFound desc = could not find container \"f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41\": container with ID starting with f1432b416468b060e1f9eafcf67b23c8cda8c6aead9d2418e4d2984b814a0b41 not found: ID does not exist"
Oct 30 13:55:47.928337 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.928294    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"]
Oct 30 13:55:47.940737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:47.940692    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f49b40/rate-test-model-f49b40-kserve-7bd76b4bdf-rxqx4"]
Oct 30 13:55:48.805411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:48.805362    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"]
Oct 30 13:55:48.805954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:48.805646    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy" containerID="cri-o://7ed52b79d78d5e0eb7ea68c48f4efa8f748d8a56b4972d34cbe4d29e241053c1" gracePeriod=30
Oct 30 13:55:49.450013 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.449964    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"]
Oct 30 13:55:49.450526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450509    4993 cpu_manager.go:414] "containerMap: removing container" podUID="642ac614-b975-403c-ad34-dfd6074b9692" containerName="revoke-keys"
Oct 30 13:55:49.450578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450527    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642ac614-b975-403c-ad34-dfd6074b9692" containerName="revoke-keys"
Oct 30 13:55:49.450578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450544    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6059e369-d2ed-4f31-8409-624fe832c897" containerName="main"
Oct 30 13:55:49.450578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450549    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6059e369-d2ed-4f31-8409-624fe832c897" containerName="main"
Oct 30 13:55:49.450578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450568    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2e830b61-6e31-4e88-aec3-3f69597a21e7" containerName="revoke-keys"
Oct 30 13:55:49.450578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450573    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e830b61-6e31-4e88-aec3-3f69597a21e7" containerName="revoke-keys"
Oct 30 13:55:49.450740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450648    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="642ac614-b975-403c-ad34-dfd6074b9692" containerName="revoke-keys"
Oct 30 13:55:49.450740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450660    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6059e369-d2ed-4f31-8409-624fe832c897" containerName="main"
Oct 30 13:55:49.450740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.450669    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e830b61-6e31-4e88-aec3-3f69597a21e7" containerName="revoke-keys"
Oct 30 13:55:49.455544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.455505    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.472166 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.472120    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:55:49.474618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.474583    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:55:49.475298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.475274    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:55:49.477488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.477456    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"]
Oct 30 13:55:49.507416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.507378    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.507848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.507819    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.508147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.508119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjn5\" (UniqueName: \"kubernetes.io/projected/d8b8597a-87b2-4425-b22e-8cd86b479215-kube-api-access-xzjn5\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.571146 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.571100    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"]
Oct 30 13:55:49.572478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.572441    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:55:49.576383 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.576355    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.597277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.597049    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"]
Oct 30 13:55:49.609334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.609292    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.609524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.609433    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.609606 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.609537    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.609716 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.609690    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdddw\" (UniqueName: \"kubernetes.io/projected/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-kube-api-access-gdddw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.609851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.609740    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.609851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.609814    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjn5\" (UniqueName: \"kubernetes.io/projected/d8b8597a-87b2-4425-b22e-8cd86b479215-kube-api-access-xzjn5\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.610831 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.610805    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.612634 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.612607    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.634306 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.634270    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjn5\" (UniqueName: \"kubernetes.io/projected/d8b8597a-87b2-4425-b22e-8cd86b479215-kube-api-access-xzjn5\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.694478 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.694436    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6059e369-d2ed-4f31-8409-624fe832c897" path="/var/lib/kubelet/pods/6059e369-d2ed-4f31-8409-624fe832c897/volumes"
Oct 30 13:55:49.710925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.710830    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.710925 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.710892    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.711169 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.710955    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdddw\" (UniqueName: \"kubernetes.io/projected/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-kube-api-access-gdddw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.711689 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.711662    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.713489 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.713465    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.741761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.741722    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdddw\" (UniqueName: \"kubernetes.io/projected/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-kube-api-access-gdddw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:49.770646 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.770605    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:49.898928 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:49.898871    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:50.087026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:50.086992    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"]
Oct 30 13:55:50.088747 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:55:50.088717    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b8597a_87b2_4425_b22e_8cd86b479215.slice/crio-b14c6ef512649bb55ab3f12d53eda8d7119419a0ea12029491bcd56569c3702b WatchSource:0}: Error finding container b14c6ef512649bb55ab3f12d53eda8d7119419a0ea12029491bcd56569c3702b: Status 404 returned error can't find the container with id b14c6ef512649bb55ab3f12d53eda8d7119419a0ea12029491bcd56569c3702b
Oct 30 13:55:50.221859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:50.221826    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"]
Oct 30 13:55:50.222819 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:55:50.222786    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f82ec2e_d6ac_43fc_9e84_0fc7b5843632.slice/crio-7d5d9b64a9b0d067d1ec5bf8ab74baf10fb8439ca111d43237946f2d7f6ec390 WatchSource:0}: Error finding container 7d5d9b64a9b0d067d1ec5bf8ab74baf10fb8439ca111d43237946f2d7f6ec390: Status 404 returned error can't find the container with id 7d5d9b64a9b0d067d1ec5bf8ab74baf10fb8439ca111d43237946f2d7f6ec390
Oct 30 13:55:50.908457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:50.908341    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42" event={"ID":"d8b8597a-87b2-4425-b22e-8cd86b479215","Type":"ContainerStarted","Data":"060a433435908ab75dc088f3d7613616fa683d3a2cfd09030c7fede341a46ac2"}
Oct 30 13:55:50.908457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:50.908413    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42" event={"ID":"d8b8597a-87b2-4425-b22e-8cd86b479215","Type":"ContainerStarted","Data":"b14c6ef512649bb55ab3f12d53eda8d7119419a0ea12029491bcd56569c3702b"}
Oct 30 13:55:50.910433 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:50.910396    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7" event={"ID":"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632","Type":"ContainerStarted","Data":"644d746629ad576f60d63c2b7f149b387a0ffce7ca5dd567d5246b05fdc3773d"}
Oct 30 13:55:50.910571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:50.910457    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7" event={"ID":"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632","Type":"ContainerStarted","Data":"7d5d9b64a9b0d067d1ec5bf8ab74baf10fb8439ca111d43237946f2d7f6ec390"}
Oct 30 13:55:50.951151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:50.951086    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42" podStartSLOduration=1.951073082 podStartE2EDuration="1.951073082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:50.948109229 +0000 UTC m=+5015.423828840" watchObservedRunningTime="2026-10-30 13:55:50.951073082 +0000 UTC m=+5015.426792693"
Oct 30 13:55:50.992501 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:50.992437    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7" podStartSLOduration=1.99242042 podStartE2EDuration="1.99242042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:50.988846764 +0000 UTC m=+5015.464566374" watchObservedRunningTime="2026-10-30 13:55:50.99242042 +0000 UTC m=+5015.468140024"
Oct 30 13:55:51.919283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:51.919199    4993 generic.go:356] "Generic (PLEG): container finished" podID="3f82ec2e-d6ac-43fc-9e84-0fc7b5843632" containerID="644d746629ad576f60d63c2b7f149b387a0ffce7ca5dd567d5246b05fdc3773d" exitCode=0
Oct 30 13:55:51.919730 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:51.919281    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7" event={"ID":"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632","Type":"ContainerDied","Data":"644d746629ad576f60d63c2b7f149b387a0ffce7ca5dd567d5246b05fdc3773d"}
Oct 30 13:55:51.921254 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:51.921226    4993 generic.go:356] "Generic (PLEG): container finished" podID="d8b8597a-87b2-4425-b22e-8cd86b479215" containerID="060a433435908ab75dc088f3d7613616fa683d3a2cfd09030c7fede341a46ac2" exitCode=0
Oct 30 13:55:51.921404 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:51.921314    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42" event={"ID":"d8b8597a-87b2-4425-b22e-8cd86b479215","Type":"ContainerDied","Data":"060a433435908ab75dc088f3d7613616fa683d3a2cfd09030c7fede341a46ac2"}
Oct 30 13:55:53.098221 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.098176    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:53.149980 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.149938    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:53.154184 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.154153    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-openshift-ca\") pod \"d8b8597a-87b2-4425-b22e-8cd86b479215\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") "
Oct 30 13:55:53.154383 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.154229    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-xks-ca\") pod \"d8b8597a-87b2-4425-b22e-8cd86b479215\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") "
Oct 30 13:55:53.154383 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.154329    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b8597a-87b2-4425-b22e-8cd86b479215-kube-api-access-xzjn5\" (UniqueName: \"kubernetes.io/projected/d8b8597a-87b2-4425-b22e-8cd86b479215-kube-api-access-xzjn5\") pod \"d8b8597a-87b2-4425-b22e-8cd86b479215\" (UID: \"d8b8597a-87b2-4425-b22e-8cd86b479215\") "
Oct 30 13:55:53.154599 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.154570    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-openshift-ca" pod "d8b8597a-87b2-4425-b22e-8cd86b479215" (UID: "d8b8597a-87b2-4425-b22e-8cd86b479215"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:53.154769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.154752    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:53.156740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.156703    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-xks-ca" pod "d8b8597a-87b2-4425-b22e-8cd86b479215" (UID: "d8b8597a-87b2-4425-b22e-8cd86b479215"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:53.157125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.157095    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b8597a-87b2-4425-b22e-8cd86b479215-kube-api-access-xzjn5" pod "d8b8597a-87b2-4425-b22e-8cd86b479215" (UID: "d8b8597a-87b2-4425-b22e-8cd86b479215"). InnerVolumeSpecName "kube-api-access-xzjn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:53.255886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.255840    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-openshift-ca\") pod \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") "
Oct 30 13:55:53.256112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.255902    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-xks-ca\") pod \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") "
Oct 30 13:55:53.256112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.256064    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-kube-api-access-gdddw\" (UniqueName: \"kubernetes.io/projected/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-kube-api-access-gdddw\") pod \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\" (UID: \"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632\") "
Oct 30 13:55:53.256644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.256591    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-openshift-ca" pod "3f82ec2e-d6ac-43fc-9e84-0fc7b5843632" (UID: "3f82ec2e-d6ac-43fc-9e84-0fc7b5843632"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:53.256821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.256684    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8597a-87b2-4425-b22e-8cd86b479215-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:53.256821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.256704    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzjn5\" (UniqueName: \"kubernetes.io/projected/d8b8597a-87b2-4425-b22e-8cd86b479215-kube-api-access-xzjn5\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:53.256821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.256714    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:53.259596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.259557    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-xks-ca" pod "3f82ec2e-d6ac-43fc-9e84-0fc7b5843632" (UID: "3f82ec2e-d6ac-43fc-9e84-0fc7b5843632"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:53.261053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.261027    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-kube-api-access-gdddw" pod "3f82ec2e-d6ac-43fc-9e84-0fc7b5843632" (UID: "3f82ec2e-d6ac-43fc-9e84-0fc7b5843632"). InnerVolumeSpecName "kube-api-access-gdddw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:53.375541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.372179    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdddw\" (UniqueName: \"kubernetes.io/projected/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-kube-api-access-gdddw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:53.375541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.372240    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:53.940903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.940866    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"
Oct 30 13:55:53.940903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.940885    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42" event={"ID":"d8b8597a-87b2-4425-b22e-8cd86b479215","Type":"ContainerDied","Data":"b14c6ef512649bb55ab3f12d53eda8d7119419a0ea12029491bcd56569c3702b"}
Oct 30 13:55:53.941256 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.940950    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14c6ef512649bb55ab3f12d53eda8d7119419a0ea12029491bcd56569c3702b"
Oct 30 13:55:53.942798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.942761    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"
Oct 30 13:55:53.942798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.942781    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7" event={"ID":"3f82ec2e-d6ac-43fc-9e84-0fc7b5843632","Type":"ContainerDied","Data":"7d5d9b64a9b0d067d1ec5bf8ab74baf10fb8439ca111d43237946f2d7f6ec390"}
Oct 30 13:55:53.943012 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.942839    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d5d9b64a9b0d067d1ec5bf8ab74baf10fb8439ca111d43237946f2d7f6ec390"
Oct 30 13:55:53.949934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.949900    4993 generic.go:356] "Generic (PLEG): container finished" podID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerID="7ed52b79d78d5e0eb7ea68c48f4efa8f748d8a56b4972d34cbe4d29e241053c1" exitCode=0
Oct 30 13:55:53.950127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.949987    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" event={"ID":"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea","Type":"ContainerDied","Data":"7ed52b79d78d5e0eb7ea68c48f4efa8f748d8a56b4972d34cbe4d29e241053c1"}
Oct 30 13:55:53.950127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:53.950064    4993 scope.go:122] "RemoveContainer" containerID="7619a3a4b739d7972eb1429f3403df469040cece3176b68783607b9428e205a8"
Oct 30 13:55:54.125736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.125701    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.288746    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-token\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-token\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.288841    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-socket\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.288947    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-certs\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.289017    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-data\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.289053    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-kube-api-access-jrfhp\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-kube-api-access-jrfhp\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.289080    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-podinfo\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.289112    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-envoy\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.289131    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istiod-ca-cert\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.289618 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.289173    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-credential-socket\") pod \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\" (UID: \"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea\") "
Oct 30 13:55:54.290644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.289867    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-credential-socket" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.290644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.290188    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istiod-ca-cert" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:54.290644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.290472    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-certs" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.290848 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.290773    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-socket" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.291019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.290995    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-data" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.293513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.293467    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-token" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:54.294568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.294539    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-podinfo" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:54.295382 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.295352    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-envoy" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.295684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.295657    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-kube-api-access-jrfhp" pod "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" (UID: "950a834d-9d0c-4b93-bcd6-1ef58e5b0bea"). InnerVolumeSpecName "kube-api-access-jrfhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:54.391103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391048    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391087    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391098    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391106    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrfhp\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-kube-api-access-jrfhp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391103 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391116    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391125    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391133    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391140    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.391149    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.959998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.959959    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"
Oct 30 13:55:54.960239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.960001    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7" event={"ID":"950a834d-9d0c-4b93-bcd6-1ef58e5b0bea","Type":"ContainerDied","Data":"febf8726864f42130e0e2bd47636453cbebebd3bfd2079d19b75c068216d257a"}
Oct 30 13:55:54.960239 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:54.960073    4993 scope.go:122] "RemoveContainer" containerID="7ed52b79d78d5e0eb7ea68c48f4efa8f748d8a56b4972d34cbe4d29e241053c1"
Oct 30 13:55:55.017727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:55.017687    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"]
Oct 30 13:55:55.029590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:55.029550    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-xczk7"]
Oct 30 13:55:55.701867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:55.701825    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" path="/var/lib/kubelet/pods/950a834d-9d0c-4b93-bcd6-1ef58e5b0bea/volumes"
Oct 30 13:55:56.229826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:56.229777    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"]
Oct 30 13:55:56.241936 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:56.241894    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4db150-nh56g"]
Oct 30 13:55:57.691309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:55:57.691268    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e830b61-6e31-4e88-aec3-3f69597a21e7" path="/var/lib/kubelet/pods/2e830b61-6e31-4e88-aec3-3f69597a21e7/volumes"
Oct 30 13:56:02.340019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.339949    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main" containerID="cri-o://527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe" gracePeriod=45
Oct 30 13:56:02.367917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.367847    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main" containerID="cri-o://94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425" gracePeriod=45
Oct 30 13:56:02.653185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.653144    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 13:56:02.681392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.681357    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 13:56:02.809049 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.808990    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-dshm\") pod \"0de7529f-c724-4629-b333-2a1eb259794d\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") "
Oct 30 13:56:02.809307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809081    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-model-cache\") pod \"0de7529f-c724-4629-b333-2a1eb259794d\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") "
Oct 30 13:56:02.809307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809116    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-home\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-home\") pod \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") "
Oct 30 13:56:02.809307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809192    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-dshm\") pod \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") "
Oct 30 13:56:02.809307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809239    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de7529f-c724-4629-b333-2a1eb259794d-kube-api-access-vr9x8\" (UniqueName: \"kubernetes.io/projected/0de7529f-c724-4629-b333-2a1eb259794d-kube-api-access-vr9x8\") pod \"0de7529f-c724-4629-b333-2a1eb259794d\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") "
Oct 30 13:56:02.809307 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809287    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de7529f-c724-4629-b333-2a1eb259794d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0de7529f-c724-4629-b333-2a1eb259794d-tls-certs\") pod \"0de7529f-c724-4629-b333-2a1eb259794d\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") "
Oct 30 13:56:02.809626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809317    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-kube-api-access-pkg96\" (UniqueName: \"kubernetes.io/projected/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-kube-api-access-pkg96\") pod \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") "
Oct 30 13:56:02.809626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809361    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-tmp-dir\") pod \"0de7529f-c724-4629-b333-2a1eb259794d\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") "
Oct 30 13:56:02.809626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809460    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-home\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-home\") pod \"0de7529f-c724-4629-b333-2a1eb259794d\" (UID: \"0de7529f-c724-4629-b333-2a1eb259794d\") "
Oct 30 13:56:02.809626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809505    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-home" pod "cbc2aea1-feb7-4a82-af41-b9173dcf6c71" (UID: "cbc2aea1-feb7-4a82-af41-b9173dcf6c71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.809626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809519    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tmp-dir\") pod \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") "
Oct 30 13:56:02.809626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809581    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tls-certs\") pod \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") "
Oct 30 13:56:02.809626 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809626    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-model-cache\") pod \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\" (UID: \"cbc2aea1-feb7-4a82-af41-b9173dcf6c71\") "
Oct 30 13:56:02.810030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809728    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tmp-dir" pod "cbc2aea1-feb7-4a82-af41-b9173dcf6c71" (UID: "cbc2aea1-feb7-4a82-af41-b9173dcf6c71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.810030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.809860    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-home" pod "0de7529f-c724-4629-b333-2a1eb259794d" (UID: "0de7529f-c724-4629-b333-2a1eb259794d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.810123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.810082    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-model-cache" pod "cbc2aea1-feb7-4a82-af41-b9173dcf6c71" (UID: "cbc2aea1-feb7-4a82-af41-b9173dcf6c71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.810123 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.810090    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-tmp-dir" pod "0de7529f-c724-4629-b333-2a1eb259794d" (UID: "0de7529f-c724-4629-b333-2a1eb259794d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.810659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.810636    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.810659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.810659    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.810840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.810668    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.810840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.810677    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.810840 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.810686    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.811536 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.811509    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-model-cache" pod "0de7529f-c724-4629-b333-2a1eb259794d" (UID: "0de7529f-c724-4629-b333-2a1eb259794d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.812249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.812186    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-dshm" pod "0de7529f-c724-4629-b333-2a1eb259794d" (UID: "0de7529f-c724-4629-b333-2a1eb259794d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.812354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.812308    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-kube-api-access-pkg96" pod "cbc2aea1-feb7-4a82-af41-b9173dcf6c71" (UID: "cbc2aea1-feb7-4a82-af41-b9173dcf6c71"). InnerVolumeSpecName "kube-api-access-pkg96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:02.812971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.812945    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tls-certs" pod "cbc2aea1-feb7-4a82-af41-b9173dcf6c71" (UID: "cbc2aea1-feb7-4a82-af41-b9173dcf6c71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:02.813086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.813062    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-dshm" pod "cbc2aea1-feb7-4a82-af41-b9173dcf6c71" (UID: "cbc2aea1-feb7-4a82-af41-b9173dcf6c71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.813823 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.813794    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de7529f-c724-4629-b333-2a1eb259794d-kube-api-access-vr9x8" pod "0de7529f-c724-4629-b333-2a1eb259794d" (UID: "0de7529f-c724-4629-b333-2a1eb259794d"). InnerVolumeSpecName "kube-api-access-vr9x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:02.813918 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.813796    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de7529f-c724-4629-b333-2a1eb259794d-tls-certs" pod "0de7529f-c724-4629-b333-2a1eb259794d" (UID: "0de7529f-c724-4629-b333-2a1eb259794d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:02.912011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.911908    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.912011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.911949    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de7529f-c724-4629-b333-2a1eb259794d-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.912011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.911961    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.912011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.911970    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9x8\" (UniqueName: \"kubernetes.io/projected/0de7529f-c724-4629-b333-2a1eb259794d-kube-api-access-vr9x8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.912011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.911979    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de7529f-c724-4629-b333-2a1eb259794d-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.912011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.911990    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkg96\" (UniqueName: \"kubernetes.io/projected/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-kube-api-access-pkg96\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.912011 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:02.912001    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc2aea1-feb7-4a82-af41-b9173dcf6c71-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:03.122094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.122053    4993 generic.go:356] "Generic (PLEG): container finished" podID="0de7529f-c724-4629-b333-2a1eb259794d" containerID="94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425" exitCode=0
Oct 30 13:56:03.122339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.122106    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"
Oct 30 13:56:03.122339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.122143    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" event={"ID":"0de7529f-c724-4629-b333-2a1eb259794d","Type":"ContainerDied","Data":"94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425"}
Oct 30 13:56:03.122339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.122196    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh" event={"ID":"0de7529f-c724-4629-b333-2a1eb259794d","Type":"ContainerDied","Data":"59d6cad21d5fee2af5e7e7156bbefa149826a39c8268190d9664acf17b19624a"}
Oct 30 13:56:03.122339 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.122234    4993 scope.go:122] "RemoveContainer" containerID="94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425"
Oct 30 13:56:03.123965 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.123927    4993 generic.go:356] "Generic (PLEG): container finished" podID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerID="527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe" exitCode=0
Oct 30 13:56:03.124086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.123985    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"
Oct 30 13:56:03.124086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.123989    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" event={"ID":"cbc2aea1-feb7-4a82-af41-b9173dcf6c71","Type":"ContainerDied","Data":"527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe"}
Oct 30 13:56:03.124086 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.124032    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh" event={"ID":"cbc2aea1-feb7-4a82-af41-b9173dcf6c71","Type":"ContainerDied","Data":"e24d3604919ea2ec654d8bc834dd7a5c33c5812995da5d22f1c1c8a7fd83e003"}
Oct 30 13:56:03.134843 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.134818    4993 scope.go:122] "RemoveContainer" containerID="94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425"
Oct 30 13:56:03.135174 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:56:03.135154    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425\": container with ID starting with 94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425 not found: ID does not exist" containerID="94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425"
Oct 30 13:56:03.135270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.135186    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425"} err="failed to get container status \"94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425\": rpc error: code = NotFound desc = could not find container \"94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425\": container with ID starting with 94a3891224163e75b2f5363b121271d8630ac2dee5267b5aabffc9245cabe425 not found: ID does not exist"
Oct 30 13:56:03.135270 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.135217    4993 scope.go:122] "RemoveContainer" containerID="527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe"
Oct 30 13:56:03.146423 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.146399    4993 scope.go:122] "RemoveContainer" containerID="527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe"
Oct 30 13:56:03.146747 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:56:03.146723    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe\": container with ID starting with 527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe not found: ID does not exist" containerID="527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe"
Oct 30 13:56:03.146832 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.146755    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe"} err="failed to get container status \"527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe\": rpc error: code = NotFound desc = could not find container \"527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe\": container with ID starting with 527545dff6362b6a271aa2495e35200a7db5987338a145113558ce48beea0ebe not found: ID does not exist"
Oct 30 13:56:03.179855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.179754    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"]
Oct 30 13:56:03.187064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.187023    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-w4-e652ca-kserve-56449f47b9-5dmmh"]
Oct 30 13:56:03.218505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.218465    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"]
Oct 30 13:56:03.222504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.222464    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/simulator-w4-e652ca-kserve-b4664f99f-nz4nh"]
Oct 30 13:56:03.693245 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.693194    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de7529f-c724-4629-b333-2a1eb259794d" path="/var/lib/kubelet/pods/0de7529f-c724-4629-b333-2a1eb259794d/volumes"
Oct 30 13:56:03.693796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:03.693653    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" path="/var/lib/kubelet/pods/cbc2aea1-feb7-4a82-af41-b9173dcf6c71/volumes"
Oct 30 13:56:04.770291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770250    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"]
Oct 30 13:56:04.770795 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770778    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main"
Oct 30 13:56:04.770861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770799    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main"
Oct 30 13:56:04.770861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770818    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d8b8597a-87b2-4425-b22e-8cd86b479215" containerName="revoke-keys"
Oct 30 13:56:04.770861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770824    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b8597a-87b2-4425-b22e-8cd86b479215" containerName="revoke-keys"
Oct 30 13:56:04.770861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770833    4993 cpu_manager.go:414] "containerMap: removing container" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy"
Oct 30 13:56:04.770861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770841    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy"
Oct 30 13:56:04.770861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770848    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main"
Oct 30 13:56:04.770861 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770852    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main"
Oct 30 13:56:04.771113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770869    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3f82ec2e-d6ac-43fc-9e84-0fc7b5843632" containerName="revoke-keys"
Oct 30 13:56:04.771113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770877    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f82ec2e-d6ac-43fc-9e84-0fc7b5843632" containerName="revoke-keys"
Oct 30 13:56:04.771113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770979    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b8597a-87b2-4425-b22e-8cd86b479215" containerName="revoke-keys"
Oct 30 13:56:04.771113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770990    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy"
Oct 30 13:56:04.771113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.770995    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc2aea1-feb7-4a82-af41-b9173dcf6c71" containerName="main"
Oct 30 13:56:04.771113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.771003    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f82ec2e-d6ac-43fc-9e84-0fc7b5843632" containerName="revoke-keys"
Oct 30 13:56:04.771113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.771011    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy"
Oct 30 13:56:04.771113 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.771019    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de7529f-c724-4629-b333-2a1eb259794d" containerName="main"
Oct 30 13:56:04.776165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.776139    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.801773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.801723    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"]
Oct 30 13:56:04.807523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.807488    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:56:04.807523 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.807488    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:56:04.846046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.846004    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.846046 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.846056    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clf2q\" (UniqueName: \"kubernetes.io/projected/45684268-12e3-403b-85a7-84d1eb776d92-kube-api-access-clf2q\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.846364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.846316    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.891054 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.891017    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:56:04.891298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.891105    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:56:04.947199 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.947152    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.947490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.947274    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.947490 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.947309    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clf2q\" (UniqueName: \"kubernetes.io/projected/45684268-12e3-403b-85a7-84d1eb776d92-kube-api-access-clf2q\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.947940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.947914    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.949857 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.949833    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:04.970568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:04.970534    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clf2q\" (UniqueName: \"kubernetes.io/projected/45684268-12e3-403b-85a7-84d1eb776d92-kube-api-access-clf2q\") pod \"maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:05.096284 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:05.093296    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:05.463524 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:05.463490    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"]
Oct 30 13:56:05.465167 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:56:05.465130    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45684268_12e3_403b_85a7_84d1eb776d92.slice/crio-4516601f7c0f673ba6daeb670693e628384d21d726befd499fc4f630c3da9222 WatchSource:0}: Error finding container 4516601f7c0f673ba6daeb670693e628384d21d726befd499fc4f630c3da9222: Status 404 returned error can't find the container with id 4516601f7c0f673ba6daeb670693e628384d21d726befd499fc4f630c3da9222
Oct 30 13:56:06.182073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.181979    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs" event={"ID":"45684268-12e3-403b-85a7-84d1eb776d92","Type":"ContainerStarted","Data":"44bef4ab95b3ae00eed8dfccccf1c779eef5fa63daf2d02d9a017bf63b4e0e7e"}
Oct 30 13:56:06.182073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.182041    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs" event={"ID":"45684268-12e3-403b-85a7-84d1eb776d92","Type":"ContainerStarted","Data":"4516601f7c0f673ba6daeb670693e628384d21d726befd499fc4f630c3da9222"}
Oct 30 13:56:06.224688 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.224621    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs" podStartSLOduration=2.224606993 podStartE2EDuration="2.224606993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:06.221598853 +0000 UTC m=+5030.697318463" watchObservedRunningTime="2026-10-30 13:56:06.224606993 +0000 UTC m=+5030.700326604"
Oct 30 13:56:06.368862 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.368829    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"]
Oct 30 13:56:06.369383 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.369364    4993 cpu_manager.go:414] "containerMap: removing container" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy"
Oct 30 13:56:06.369485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.369384    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950a834d-9d0c-4b93-bcd6-1ef58e5b0bea" containerName="istio-proxy"
Oct 30 13:56:06.372877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.372856    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.395799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.395756    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"]
Oct 30 13:56:06.467219 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.467164    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/eca05d59-a006-4ff1-a844-3edd9761878b-kube-api-access-sphtw\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.467396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.467324    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.467396 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.467359    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.568782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.568726    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/eca05d59-a006-4ff1-a844-3edd9761878b-kube-api-access-sphtw\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.568985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.568829    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.568985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.568865    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.569762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.569724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.572100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.572067    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.592826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.592792    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/eca05d59-a006-4ff1-a844-3edd9761878b-kube-api-access-sphtw\") pod \"maas-api-revoke-sub-bf8ccd0e-lh9vh\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:06.746585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:06.744772    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:07.064251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:07.063340    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"]
Oct 30 13:56:07.064251 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:56:07.063494    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca05d59_a006_4ff1_a844_3edd9761878b.slice/crio-67580086f3c9f74fb948becbd5e2fc08262a6e5647750100cbdc1f95bd14bac6 WatchSource:0}: Error finding container 67580086f3c9f74fb948becbd5e2fc08262a6e5647750100cbdc1f95bd14bac6: Status 404 returned error can't find the container with id 67580086f3c9f74fb948becbd5e2fc08262a6e5647750100cbdc1f95bd14bac6
Oct 30 13:56:07.192692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:07.192634    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh" event={"ID":"eca05d59-a006-4ff1-a844-3edd9761878b","Type":"ContainerStarted","Data":"67580086f3c9f74fb948becbd5e2fc08262a6e5647750100cbdc1f95bd14bac6"}
Oct 30 13:56:07.194656 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:07.194628    4993 generic.go:356] "Generic (PLEG): container finished" podID="45684268-12e3-403b-85a7-84d1eb776d92" containerID="44bef4ab95b3ae00eed8dfccccf1c779eef5fa63daf2d02d9a017bf63b4e0e7e" exitCode=0
Oct 30 13:56:07.194811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:07.194713    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs" event={"ID":"45684268-12e3-403b-85a7-84d1eb776d92","Type":"ContainerDied","Data":"44bef4ab95b3ae00eed8dfccccf1c779eef5fa63daf2d02d9a017bf63b4e0e7e"}
Oct 30 13:56:07.224968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:07.224920    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"]
Oct 30 13:56:07.228467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:07.228433    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92855c-zr7k7"]
Oct 30 13:56:07.691584 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:07.691500    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f82ec2e-d6ac-43fc-9e84-0fc7b5843632" path="/var/lib/kubelet/pods/3f82ec2e-d6ac-43fc-9e84-0fc7b5843632/volumes"
Oct 30 13:56:08.205392 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.205356    4993 generic.go:356] "Generic (PLEG): container finished" podID="eca05d59-a006-4ff1-a844-3edd9761878b" containerID="8fe099479423fea406c31607123ec411335137444772fb9a6bdada1ed050cc18" exitCode=0
Oct 30 13:56:08.205851 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.205442    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh" event={"ID":"eca05d59-a006-4ff1-a844-3edd9761878b","Type":"ContainerDied","Data":"8fe099479423fea406c31607123ec411335137444772fb9a6bdada1ed050cc18"}
Oct 30 13:56:08.357365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.357333    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:08.499948 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.499901    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-openshift-ca\") pod \"45684268-12e3-403b-85a7-84d1eb776d92\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") "
Oct 30 13:56:08.500148 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.500007    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-xks-ca\") pod \"45684268-12e3-403b-85a7-84d1eb776d92\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") "
Oct 30 13:56:08.500259 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.500191    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45684268-12e3-403b-85a7-84d1eb776d92-kube-api-access-clf2q\" (UniqueName: \"kubernetes.io/projected/45684268-12e3-403b-85a7-84d1eb776d92-kube-api-access-clf2q\") pod \"45684268-12e3-403b-85a7-84d1eb776d92\" (UID: \"45684268-12e3-403b-85a7-84d1eb776d92\") "
Oct 30 13:56:08.500387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.500355    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-openshift-ca" pod "45684268-12e3-403b-85a7-84d1eb776d92" (UID: "45684268-12e3-403b-85a7-84d1eb776d92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:08.500643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.500621    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:08.502826 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.502799    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45684268-12e3-403b-85a7-84d1eb776d92-kube-api-access-clf2q" pod "45684268-12e3-403b-85a7-84d1eb776d92" (UID: "45684268-12e3-403b-85a7-84d1eb776d92"). InnerVolumeSpecName "kube-api-access-clf2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:08.502967 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.502884    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-xks-ca" pod "45684268-12e3-403b-85a7-84d1eb776d92" (UID: "45684268-12e3-403b-85a7-84d1eb776d92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:08.579696 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:56:08.579599    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:56:08.601904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.601859    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45684268-12e3-403b-85a7-84d1eb776d92-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:08.601904 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:08.601904    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clf2q\" (UniqueName: \"kubernetes.io/projected/45684268-12e3-403b-85a7-84d1eb776d92-kube-api-access-clf2q\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:09.215503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.215468    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"
Oct 30 13:56:09.215503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.215494    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs" event={"ID":"45684268-12e3-403b-85a7-84d1eb776d92","Type":"ContainerDied","Data":"4516601f7c0f673ba6daeb670693e628384d21d726befd499fc4f630c3da9222"}
Oct 30 13:56:09.216015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.215555    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4516601f7c0f673ba6daeb670693e628384d21d726befd499fc4f630c3da9222"
Oct 30 13:56:09.381316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.381288    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:09.410472 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.410428    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-xks-ca\") pod \"eca05d59-a006-4ff1-a844-3edd9761878b\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") "
Oct 30 13:56:09.410665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.410559    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca05d59-a006-4ff1-a844-3edd9761878b-kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/eca05d59-a006-4ff1-a844-3edd9761878b-kube-api-access-sphtw\") pod \"eca05d59-a006-4ff1-a844-3edd9761878b\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") "
Oct 30 13:56:09.410665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.410613    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-openshift-ca\") pod \"eca05d59-a006-4ff1-a844-3edd9761878b\" (UID: \"eca05d59-a006-4ff1-a844-3edd9761878b\") "
Oct 30 13:56:09.411017 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.410991    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-openshift-ca" pod "eca05d59-a006-4ff1-a844-3edd9761878b" (UID: "eca05d59-a006-4ff1-a844-3edd9761878b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:09.412758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.412733    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-xks-ca" pod "eca05d59-a006-4ff1-a844-3edd9761878b" (UID: "eca05d59-a006-4ff1-a844-3edd9761878b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:09.412888 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.412867    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca05d59-a006-4ff1-a844-3edd9761878b-kube-api-access-sphtw" pod "eca05d59-a006-4ff1-a844-3edd9761878b" (UID: "eca05d59-a006-4ff1-a844-3edd9761878b"). InnerVolumeSpecName "kube-api-access-sphtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:09.511622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.511589    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/eca05d59-a006-4ff1-a844-3edd9761878b-kube-api-access-sphtw\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:09.511622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.511620    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:09.511833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:09.511631    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca05d59-a006-4ff1-a844-3edd9761878b-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:10.225860 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:10.225824    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh"
Oct 30 13:56:10.226321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:10.225818    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8ccd0e-lh9vh" event={"ID":"eca05d59-a006-4ff1-a844-3edd9761878b","Type":"ContainerDied","Data":"67580086f3c9f74fb948becbd5e2fc08262a6e5647750100cbdc1f95bd14bac6"}
Oct 30 13:56:10.226321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:10.225990    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67580086f3c9f74fb948becbd5e2fc08262a6e5647750100cbdc1f95bd14bac6"
Oct 30 13:56:11.246397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:11.246356    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"]
Oct 30 13:56:11.278808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:11.278762    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-517a25-ddf42"]
Oct 30 13:56:11.693825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:11.693732    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b8597a-87b2-4425-b22e-8cd86b479215" path="/var/lib/kubelet/pods/d8b8597a-87b2-4425-b22e-8cd86b479215/volumes"
Oct 30 13:56:15.704694 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.704655    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"]
Oct 30 13:56:15.705387 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.705366    4993 cpu_manager.go:414] "containerMap: removing container" podUID="45684268-12e3-403b-85a7-84d1eb776d92" containerName="revoke-keys"
Oct 30 13:56:15.705447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.705390    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45684268-12e3-403b-85a7-84d1eb776d92" containerName="revoke-keys"
Oct 30 13:56:15.705447 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.705440    4993 cpu_manager.go:414] "containerMap: removing container" podUID="eca05d59-a006-4ff1-a844-3edd9761878b" containerName="revoke-keys"
Oct 30 13:56:15.705520 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.705448    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca05d59-a006-4ff1-a844-3edd9761878b" containerName="revoke-keys"
Oct 30 13:56:15.705604 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.705591    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca05d59-a006-4ff1-a844-3edd9761878b" containerName="revoke-keys"
Oct 30 13:56:15.705637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.705612    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="45684268-12e3-403b-85a7-84d1eb776d92" containerName="revoke-keys"
Oct 30 13:56:15.711029 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.711006    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.728786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.728747    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"]
Oct 30 13:56:15.736248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.736201    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:56:15.736448 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.736333    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:56:15.738578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.738527    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:56:15.738747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.738703    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:56:15.797157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.797117    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.797157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.797159    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.797440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.797179    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwcds\" (UniqueName: \"kubernetes.io/projected/b138ed9a-40b7-4d83-ad58-90d8e20deddb-kube-api-access-gwcds\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.908159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.908113    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.908498 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.908457    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.911061 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.911010    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcds\" (UniqueName: \"kubernetes.io/projected/b138ed9a-40b7-4d83-ad58-90d8e20deddb-kube-api-access-gwcds\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.911287 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.910804    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.937934 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.937847    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:15.947094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:15.947055    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcds\" (UniqueName: \"kubernetes.io/projected/b138ed9a-40b7-4d83-ad58-90d8e20deddb-kube-api-access-gwcds\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:16.038793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:16.038748    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:56:16.046095 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:16.046066    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:16.362070 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:16.362038    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"]
Oct 30 13:56:16.364242 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:56:16.364177    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb138ed9a_40b7_4d83_ad58_90d8e20deddb.slice/crio-0b26fcbdf0f0ef7ecc26f7234ab3e90a10963c845520e740dbe953a5efb3fd60 WatchSource:0}: Error finding container 0b26fcbdf0f0ef7ecc26f7234ab3e90a10963c845520e740dbe953a5efb3fd60: Status 404 returned error can't find the container with id 0b26fcbdf0f0ef7ecc26f7234ab3e90a10963c845520e740dbe953a5efb3fd60
Oct 30 13:56:17.287125 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:17.287083    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s" event={"ID":"b138ed9a-40b7-4d83-ad58-90d8e20deddb","Type":"ContainerStarted","Data":"83edc0a43c62038fcf1d016e038de769ad22c1bd3a199fd1725ac463045fdf34"}
Oct 30 13:56:17.287619 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:17.287148    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s" event={"ID":"b138ed9a-40b7-4d83-ad58-90d8e20deddb","Type":"ContainerStarted","Data":"0b26fcbdf0f0ef7ecc26f7234ab3e90a10963c845520e740dbe953a5efb3fd60"}
Oct 30 13:56:17.328393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:17.328328    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s" podStartSLOduration=2.328311466 podStartE2EDuration="2.328311466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:17.327148016 +0000 UTC m=+5041.802867627" watchObservedRunningTime="2026-10-30 13:56:17.328311466 +0000 UTC m=+5041.804031075"
Oct 30 13:56:18.298155 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:18.298112    4993 generic.go:356] "Generic (PLEG): container finished" podID="b138ed9a-40b7-4d83-ad58-90d8e20deddb" containerID="83edc0a43c62038fcf1d016e038de769ad22c1bd3a199fd1725ac463045fdf34" exitCode=0
Oct 30 13:56:18.298611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:18.298198    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s" event={"ID":"b138ed9a-40b7-4d83-ad58-90d8e20deddb","Type":"ContainerDied","Data":"83edc0a43c62038fcf1d016e038de769ad22c1bd3a199fd1725ac463045fdf34"}
Oct 30 13:56:19.464770 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.464731    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:19.564565 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.564459    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-xks-ca\") pod \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") "
Oct 30 13:56:19.564744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.564595    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-openshift-ca\") pod \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") "
Oct 30 13:56:19.564901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.564878    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b138ed9a-40b7-4d83-ad58-90d8e20deddb-kube-api-access-gwcds\" (UniqueName: \"kubernetes.io/projected/b138ed9a-40b7-4d83-ad58-90d8e20deddb-kube-api-access-gwcds\") pod \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\" (UID: \"b138ed9a-40b7-4d83-ad58-90d8e20deddb\") "
Oct 30 13:56:19.565143 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.565121    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-openshift-ca" pod "b138ed9a-40b7-4d83-ad58-90d8e20deddb" (UID: "b138ed9a-40b7-4d83-ad58-90d8e20deddb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:19.565376 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.565353    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:19.567032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.567000    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-xks-ca" pod "b138ed9a-40b7-4d83-ad58-90d8e20deddb" (UID: "b138ed9a-40b7-4d83-ad58-90d8e20deddb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:19.567159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.567106    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b138ed9a-40b7-4d83-ad58-90d8e20deddb-kube-api-access-gwcds" pod "b138ed9a-40b7-4d83-ad58-90d8e20deddb" (UID: "b138ed9a-40b7-4d83-ad58-90d8e20deddb"). InnerVolumeSpecName "kube-api-access-gwcds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:19.602544 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.602504    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"]
Oct 30 13:56:19.603064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.603043    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b138ed9a-40b7-4d83-ad58-90d8e20deddb" containerName="revoke-keys"
Oct 30 13:56:19.603064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.603061    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b138ed9a-40b7-4d83-ad58-90d8e20deddb" containerName="revoke-keys"
Oct 30 13:56:19.603251 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.603195    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b138ed9a-40b7-4d83-ad58-90d8e20deddb" containerName="revoke-keys"
Oct 30 13:56:19.606733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.606709    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.629717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.629673    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"]
Oct 30 13:56:19.666430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.666387    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwcds\" (UniqueName: \"kubernetes.io/projected/b138ed9a-40b7-4d83-ad58-90d8e20deddb-kube-api-access-gwcds\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:19.666430 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.666428    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b138ed9a-40b7-4d83-ad58-90d8e20deddb-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:19.767880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.767842    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vv4z\" (UniqueName: \"kubernetes.io/projected/9945ecb8-daa4-4e7f-9593-9300220941f2-kube-api-access-6vv4z\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.768043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.767899    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.768043 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.768003    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.869748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.869651    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vv4z\" (UniqueName: \"kubernetes.io/projected/9945ecb8-daa4-4e7f-9593-9300220941f2-kube-api-access-6vv4z\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.869748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.869719    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.869748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.869746    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.870497 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.870467    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.872390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.872358    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.892783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.892744    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vv4z\" (UniqueName: \"kubernetes.io/projected/9945ecb8-daa4-4e7f-9593-9300220941f2-kube-api-access-6vv4z\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:19.923719 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:19.923679    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:20.242281 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:20.242238    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"]
Oct 30 13:56:20.243609 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:56:20.243581    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9945ecb8_daa4_4e7f_9593_9300220941f2.slice/crio-d4d5179d1f82a496911757a96b5bb659b8425b3eb2bf0e0ad47a33bb8d6d375b WatchSource:0}: Error finding container d4d5179d1f82a496911757a96b5bb659b8425b3eb2bf0e0ad47a33bb8d6d375b: Status 404 returned error can't find the container with id d4d5179d1f82a496911757a96b5bb659b8425b3eb2bf0e0ad47a33bb8d6d375b
Oct 30 13:56:20.314703 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:20.314665    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"
Oct 30 13:56:20.314903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:20.314700    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s" event={"ID":"b138ed9a-40b7-4d83-ad58-90d8e20deddb","Type":"ContainerDied","Data":"0b26fcbdf0f0ef7ecc26f7234ab3e90a10963c845520e740dbe953a5efb3fd60"}
Oct 30 13:56:20.314903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:20.314762    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b26fcbdf0f0ef7ecc26f7234ab3e90a10963c845520e740dbe953a5efb3fd60"
Oct 30 13:56:20.316200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:20.316170    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g" event={"ID":"9945ecb8-daa4-4e7f-9593-9300220941f2","Type":"ContainerStarted","Data":"d4d5179d1f82a496911757a96b5bb659b8425b3eb2bf0e0ad47a33bb8d6d375b"}
Oct 30 13:56:20.442715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:20.442673    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"]
Oct 30 13:56:20.465335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:20.465295    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"]
Oct 30 13:56:21.325551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:21.325515    4993 generic.go:356] "Generic (PLEG): container finished" podID="9945ecb8-daa4-4e7f-9593-9300220941f2" containerID="bdd072b01633ad8b13f18ef28a43f03df8eefe82aed0d72ea5345e123c8b2a9c" exitCode=0
Oct 30 13:56:21.325750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:21.325606    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g" event={"ID":"9945ecb8-daa4-4e7f-9593-9300220941f2","Type":"ContainerDied","Data":"bdd072b01633ad8b13f18ef28a43f03df8eefe82aed0d72ea5345e123c8b2a9c"}
Oct 30 13:56:22.338399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.338364    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"]
Oct 30 13:56:22.350048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.350014    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e652ca-d7wgx"]
Oct 30 13:56:22.550450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.550420    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:22.705610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.705515    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-openshift-ca\") pod \"9945ecb8-daa4-4e7f-9593-9300220941f2\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") "
Oct 30 13:56:22.705802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.705674    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-xks-ca\") pod \"9945ecb8-daa4-4e7f-9593-9300220941f2\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") "
Oct 30 13:56:22.705874 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.705830    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9945ecb8-daa4-4e7f-9593-9300220941f2-kube-api-access-6vv4z\" (UniqueName: \"kubernetes.io/projected/9945ecb8-daa4-4e7f-9593-9300220941f2-kube-api-access-6vv4z\") pod \"9945ecb8-daa4-4e7f-9593-9300220941f2\" (UID: \"9945ecb8-daa4-4e7f-9593-9300220941f2\") "
Oct 30 13:56:22.705938 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.705899    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-openshift-ca" pod "9945ecb8-daa4-4e7f-9593-9300220941f2" (UID: "9945ecb8-daa4-4e7f-9593-9300220941f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:22.706173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.706153    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:22.707799 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.707776    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-xks-ca" pod "9945ecb8-daa4-4e7f-9593-9300220941f2" (UID: "9945ecb8-daa4-4e7f-9593-9300220941f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:22.707913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.707893    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9945ecb8-daa4-4e7f-9593-9300220941f2-kube-api-access-6vv4z" pod "9945ecb8-daa4-4e7f-9593-9300220941f2" (UID: "9945ecb8-daa4-4e7f-9593-9300220941f2"). InnerVolumeSpecName "kube-api-access-6vv4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:22.806998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.806955    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9945ecb8-daa4-4e7f-9593-9300220941f2-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:22.806998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:22.806991    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vv4z\" (UniqueName: \"kubernetes.io/projected/9945ecb8-daa4-4e7f-9593-9300220941f2-kube-api-access-6vv4z\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:23.342822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:23.342788    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"
Oct 30 13:56:23.343364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:23.342824    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g" event={"ID":"9945ecb8-daa4-4e7f-9593-9300220941f2","Type":"ContainerDied","Data":"d4d5179d1f82a496911757a96b5bb659b8425b3eb2bf0e0ad47a33bb8d6d375b"}
Oct 30 13:56:23.343364 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:23.342881    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d5179d1f82a496911757a96b5bb659b8425b3eb2bf0e0ad47a33bb8d6d375b"
Oct 30 13:56:23.716411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:23.716314    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642ac614-b975-403c-ad34-dfd6074b9692" path="/var/lib/kubelet/pods/642ac614-b975-403c-ad34-dfd6074b9692/volumes"
Oct 30 13:56:24.247760 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:24.247723    4993 scope.go:122] "RemoveContainer" containerID="1af4e1d3514f390aa749f0c72cf64786ae6e9f40465523be34bd7d53fb8b1193"
Oct 30 13:56:24.261750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:24.261723    4993 scope.go:122] "RemoveContainer" containerID="cb4eb7051b9694cc180f952b172beb58f0ee23b780267cc2c1be14835e409dc1"
Oct 30 13:56:24.274636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:24.274605    4993 scope.go:122] "RemoveContainer" containerID="3e74de7c6f00202accd657de81822c6c7f651ab53e778f9afbc39d72111e8e9f"
Oct 30 13:56:27.345683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.345643    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"]
Oct 30 13:56:27.346402 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.346377    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9945ecb8-daa4-4e7f-9593-9300220941f2" containerName="revoke-keys"
Oct 30 13:56:27.346526 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.346405    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9945ecb8-daa4-4e7f-9593-9300220941f2" containerName="revoke-keys"
Oct 30 13:56:27.346591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.346542    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9945ecb8-daa4-4e7f-9593-9300220941f2" containerName="revoke-keys"
Oct 30 13:56:27.351702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.351678    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.371323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.371291    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:56:27.373751 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.373709    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"]
Oct 30 13:56:27.378321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.378287    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"]
Oct 30 13:56:27.378451 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.378420    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.381347 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.381307    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:56:27.382559 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.382526    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:56:27.382681 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.382656    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:56:27.417484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.417442    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.417710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.417529    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.417710 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.417620    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhdns\" (UniqueName: \"kubernetes.io/projected/20c51228-21bd-4104-8fa8-eafb0954f18b-kube-api-access-fhdns\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.428386 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.428314    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"]
Oct 30 13:56:27.518786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.518735    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mz9\" (UniqueName: \"kubernetes.io/projected/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-kube-api-access-m4mz9\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.518997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.518827    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.518997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.518891    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.518997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.518922    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhdns\" (UniqueName: \"kubernetes.io/projected/20c51228-21bd-4104-8fa8-eafb0954f18b-kube-api-access-fhdns\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.518997 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.518944    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.519458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.519430    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.520069 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.520041    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.525257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.523714    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.560720 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.560681    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhdns\" (UniqueName: \"kubernetes.io/projected/20c51228-21bd-4104-8fa8-eafb0954f18b-kube-api-access-fhdns\") pod \"maas-api-revoke-sub-df82330b-xlqmn\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.620258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.620146    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.620258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.620216    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.620506 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.620376    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mz9\" (UniqueName: \"kubernetes.io/projected/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-kube-api-access-m4mz9\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.620953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.620922    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.623110 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.623084    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.657269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.657230    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mz9\" (UniqueName: \"kubernetes.io/projected/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-kube-api-access-m4mz9\") pod \"maas-api-revoke-sub-bd0da700-sqlg9\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:27.667643 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.667541    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:27.720000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:27.719626    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:28.082858 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:56:28.082811    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c51228_21bd_4104_8fa8_eafb0954f18b.slice/crio-c2de49702f691e0b43d451c330eb8c5115cddadfc315a894d19189649ee3827d WatchSource:0}: Error finding container c2de49702f691e0b43d451c330eb8c5115cddadfc315a894d19189649ee3827d: Status 404 returned error can't find the container with id c2de49702f691e0b43d451c330eb8c5115cddadfc315a894d19189649ee3827d
Oct 30 13:56:28.083507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:28.083481    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"]
Oct 30 13:56:28.114262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:28.114231    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"]
Oct 30 13:56:28.123630 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:56:28.123604    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078c5fa3_22e6_4ef2_acc1_07fbc3b32fba.slice/crio-ad9abff639038a4bf73d0abe41fcbc2e6d22a307cdacaf416086fa6cb21b20c5 WatchSource:0}: Error finding container ad9abff639038a4bf73d0abe41fcbc2e6d22a307cdacaf416086fa6cb21b20c5: Status 404 returned error can't find the container with id ad9abff639038a4bf73d0abe41fcbc2e6d22a307cdacaf416086fa6cb21b20c5
Oct 30 13:56:28.234959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:28.234911    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"]
Oct 30 13:56:28.239778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:28.239743    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-75d385-ppqbs"]
Oct 30 13:56:28.484610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:28.484573    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9" event={"ID":"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba","Type":"ContainerStarted","Data":"ad9abff639038a4bf73d0abe41fcbc2e6d22a307cdacaf416086fa6cb21b20c5"}
Oct 30 13:56:28.486436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:28.486368    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn" event={"ID":"20c51228-21bd-4104-8fa8-eafb0954f18b","Type":"ContainerStarted","Data":"60940bbcadf5063c211c7f141702dc89bc00ffe410a8d9bbe77cf8152183c5b1"}
Oct 30 13:56:28.486436 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:28.486420    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn" event={"ID":"20c51228-21bd-4104-8fa8-eafb0954f18b","Type":"ContainerStarted","Data":"c2de49702f691e0b43d451c330eb8c5115cddadfc315a894d19189649ee3827d"}
Oct 30 13:56:28.528158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:28.528083    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn" podStartSLOduration=1.528064012 podStartE2EDuration="1.528064012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:28.527433816 +0000 UTC m=+5053.003153427" watchObservedRunningTime="2026-10-30 13:56:28.528064012 +0000 UTC m=+5053.003783622"
Oct 30 13:56:29.495453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:29.495407    4993 generic.go:356] "Generic (PLEG): container finished" podID="078c5fa3-22e6-4ef2-acc1-07fbc3b32fba" containerID="431e901c53ec83a45aff0628f9f075c689473efaff5c8d1a9fa47c51f4c18934" exitCode=0
Oct 30 13:56:29.495944 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:29.495494    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9" event={"ID":"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba","Type":"ContainerDied","Data":"431e901c53ec83a45aff0628f9f075c689473efaff5c8d1a9fa47c51f4c18934"}
Oct 30 13:56:29.497555 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:29.497523    4993 generic.go:356] "Generic (PLEG): container finished" podID="20c51228-21bd-4104-8fa8-eafb0954f18b" containerID="60940bbcadf5063c211c7f141702dc89bc00ffe410a8d9bbe77cf8152183c5b1" exitCode=0
Oct 30 13:56:29.497724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:29.497552    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn" event={"ID":"20c51228-21bd-4104-8fa8-eafb0954f18b","Type":"ContainerDied","Data":"60940bbcadf5063c211c7f141702dc89bc00ffe410a8d9bbe77cf8152183c5b1"}
Oct 30 13:56:29.694403 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:29.694363    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45684268-12e3-403b-85a7-84d1eb776d92" path="/var/lib/kubelet/pods/45684268-12e3-403b-85a7-84d1eb776d92/volumes"
Oct 30 13:56:30.797434 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.797400    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:30.802665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.802632    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:30.869614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.869571    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-xks-ca\") pod \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") "
Oct 30 13:56:30.869838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.869644    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-openshift-ca\") pod \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") "
Oct 30 13:56:30.869838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.869740    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c51228-21bd-4104-8fa8-eafb0954f18b-kube-api-access-fhdns\" (UniqueName: \"kubernetes.io/projected/20c51228-21bd-4104-8fa8-eafb0954f18b-kube-api-access-fhdns\") pod \"20c51228-21bd-4104-8fa8-eafb0954f18b\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") "
Oct 30 13:56:30.869838 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.869817    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-openshift-ca\") pod \"20c51228-21bd-4104-8fa8-eafb0954f18b\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") "
Oct 30 13:56:30.870018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.869918    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-kube-api-access-m4mz9\" (UniqueName: \"kubernetes.io/projected/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-kube-api-access-m4mz9\") pod \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\" (UID: \"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba\") "
Oct 30 13:56:30.870018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.869961    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-xks-ca\") pod \"20c51228-21bd-4104-8fa8-eafb0954f18b\" (UID: \"20c51228-21bd-4104-8fa8-eafb0954f18b\") "
Oct 30 13:56:30.870174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.870145    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-openshift-ca" pod "078c5fa3-22e6-4ef2-acc1-07fbc3b32fba" (UID: "078c5fa3-22e6-4ef2-acc1-07fbc3b32fba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:30.870450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.870422    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:30.870571 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.870547    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-openshift-ca" pod "20c51228-21bd-4104-8fa8-eafb0954f18b" (UID: "20c51228-21bd-4104-8fa8-eafb0954f18b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:30.873756 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.873722    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-xks-ca" pod "20c51228-21bd-4104-8fa8-eafb0954f18b" (UID: "20c51228-21bd-4104-8fa8-eafb0954f18b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:30.874631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.874603    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-xks-ca" pod "078c5fa3-22e6-4ef2-acc1-07fbc3b32fba" (UID: "078c5fa3-22e6-4ef2-acc1-07fbc3b32fba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:30.879596 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.879556    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c51228-21bd-4104-8fa8-eafb0954f18b-kube-api-access-fhdns" pod "20c51228-21bd-4104-8fa8-eafb0954f18b" (UID: "20c51228-21bd-4104-8fa8-eafb0954f18b"). InnerVolumeSpecName "kube-api-access-fhdns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:30.885591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.884835    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-kube-api-access-m4mz9" pod "078c5fa3-22e6-4ef2-acc1-07fbc3b32fba" (UID: "078c5fa3-22e6-4ef2-acc1-07fbc3b32fba"). InnerVolumeSpecName "kube-api-access-m4mz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:30.971480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.971426    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhdns\" (UniqueName: \"kubernetes.io/projected/20c51228-21bd-4104-8fa8-eafb0954f18b-kube-api-access-fhdns\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:30.971480 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.971477    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:30.971717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.971493    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mz9\" (UniqueName: \"kubernetes.io/projected/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-kube-api-access-m4mz9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:30.971717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.971510    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c51228-21bd-4104-8fa8-eafb0954f18b-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:30.971717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:30.971525    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078c5fa3-22e6-4ef2-acc1-07fbc3b32fba-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:31.515978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:31.515945    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9"
Oct 30 13:56:31.516194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:31.515980    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0da700-sqlg9" event={"ID":"078c5fa3-22e6-4ef2-acc1-07fbc3b32fba","Type":"ContainerDied","Data":"ad9abff639038a4bf73d0abe41fcbc2e6d22a307cdacaf416086fa6cb21b20c5"}
Oct 30 13:56:31.516194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:31.516043    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9abff639038a4bf73d0abe41fcbc2e6d22a307cdacaf416086fa6cb21b20c5"
Oct 30 13:56:31.517614 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:31.517591    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn"
Oct 30 13:56:31.517758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:31.517633    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df82330b-xlqmn" event={"ID":"20c51228-21bd-4104-8fa8-eafb0954f18b","Type":"ContainerDied","Data":"c2de49702f691e0b43d451c330eb8c5115cddadfc315a894d19189649ee3827d"}
Oct 30 13:56:31.517758 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:31.517680    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2de49702f691e0b43d451c330eb8c5115cddadfc315a894d19189649ee3827d"
Oct 30 13:56:35.466693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.466644    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main" containerID="cri-o://7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf" gracePeriod=45
Oct 30 13:56:35.487834 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.487798    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" containerID="cri-o://447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe" gracePeriod=45
Oct 30 13:56:35.789019 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.788991    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:56:35.822806 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.822777    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:56:35.847442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.846604    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tmp-dir\") pod \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") "
Oct 30 13:56:35.847442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.846918    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tmp-dir" pod "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" (UID: "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:35.847442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.847117    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-kube-api-access-6w9wp\" (UniqueName: \"kubernetes.io/projected/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-kube-api-access-6w9wp\") pod \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") "
Oct 30 13:56:35.847762 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.847628    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-dshm\") pod \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") "
Oct 30 13:56:35.848130 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.848088    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-model-cache\") pod \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") "
Oct 30 13:56:35.848450 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.848410    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-model-cache" pod "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" (UID: "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:35.848636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.848617    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-home\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-home\") pod \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") "
Oct 30 13:56:35.849064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.848950    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-home" pod "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" (UID: "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:35.849583 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.849562    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tls-certs\" (UniqueName: \"kubernetes.io/secret/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tls-certs\") pod \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\" (UID: \"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d\") "
Oct 30 13:56:35.851784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.851715    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:35.851784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.851749    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:35.851784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.851764    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:35.852769 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.852730    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tls-certs" pod "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" (UID: "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:35.857845 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.857813    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-kube-api-access-6w9wp" pod "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" (UID: "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d"). InnerVolumeSpecName "kube-api-access-6w9wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:35.858106 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.858068    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-dshm" pod "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" (UID: "def5b5b9-fa1f-4d52-b32f-6ab32cbc552d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:35.952151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.952101    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-kube-api-access-p5cw8\" (UniqueName: \"kubernetes.io/projected/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-kube-api-access-p5cw8\") pod \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") "
Oct 30 13:56:35.952389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.952306    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-home\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-home\") pod \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") "
Oct 30 13:56:35.952470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.952414    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-dshm\") pod \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") "
Oct 30 13:56:35.952470 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.952459    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tls-certs\") pod \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") "
Oct 30 13:56:35.952597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.952523    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-model-cache\") pod \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") "
Oct 30 13:56:35.952597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.952573    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tmp-dir\") pod \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\" (UID: \"1f9baa7e-a484-4d2e-aeb1-238028c1a77f\") "
Oct 30 13:56:35.953151 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.952948    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:35.953641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.953622    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w9wp\" (UniqueName: \"kubernetes.io/projected/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-kube-api-access-6w9wp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:35.953766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.953747    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:35.953766 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.953521    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-model-cache" pod "1f9baa7e-a484-4d2e-aeb1-238028c1a77f" (UID: "1f9baa7e-a484-4d2e-aeb1-238028c1a77f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:35.953880 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.953758    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-home" pod "1f9baa7e-a484-4d2e-aeb1-238028c1a77f" (UID: "1f9baa7e-a484-4d2e-aeb1-238028c1a77f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:35.954315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.954282    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tmp-dir" pod "1f9baa7e-a484-4d2e-aeb1-238028c1a77f" (UID: "1f9baa7e-a484-4d2e-aeb1-238028c1a77f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:35.954955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.954930    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-dshm" pod "1f9baa7e-a484-4d2e-aeb1-238028c1a77f" (UID: "1f9baa7e-a484-4d2e-aeb1-238028c1a77f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:35.955279 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.955257    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-kube-api-access-p5cw8" pod "1f9baa7e-a484-4d2e-aeb1-238028c1a77f" (UID: "1f9baa7e-a484-4d2e-aeb1-238028c1a77f"). InnerVolumeSpecName "kube-api-access-p5cw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:35.956503 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:35.956480    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tls-certs" pod "1f9baa7e-a484-4d2e-aeb1-238028c1a77f" (UID: "1f9baa7e-a484-4d2e-aeb1-238028c1a77f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:36.054968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.054866    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:36.054968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.054900    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:36.054968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.054911    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:36.054968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.054924    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5cw8\" (UniqueName: \"kubernetes.io/projected/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-kube-api-access-p5cw8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:36.054968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.054934    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:36.054968 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.054946    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9baa7e-a484-4d2e-aeb1-238028c1a77f-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:36.343040 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.342935    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"]
Oct 30 13:56:36.343378 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.343335    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy" containerID="cri-o://42851da24d83b04cab294bcb1f4173bf18db947879ba216812821825de98934c" gracePeriod=30
Oct 30 13:56:36.558458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.558415    4993 generic.go:356] "Generic (PLEG): container finished" podID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerID="447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe" exitCode=0
Oct 30 13:56:36.558998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.558510    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"
Oct 30 13:56:36.558998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.558530    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" event={"ID":"1f9baa7e-a484-4d2e-aeb1-238028c1a77f","Type":"ContainerDied","Data":"447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe"}
Oct 30 13:56:36.558998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.558589    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" event={"ID":"1f9baa7e-a484-4d2e-aeb1-238028c1a77f","Type":"ContainerDied","Data":"433e6e47e8b4c60be345d4c20ff3463804518088646bae7e3ef710a8c2c1a3ef"}
Oct 30 13:56:36.558998 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.558616    4993 scope.go:122] "RemoveContainer" containerID="447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe"
Oct 30 13:56:36.560460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.560327    4993 generic.go:356] "Generic (PLEG): container finished" podID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerID="7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf" exitCode=0
Oct 30 13:56:36.560460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.560386    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"
Oct 30 13:56:36.560460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.560424    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" event={"ID":"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d","Type":"ContainerDied","Data":"7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf"}
Oct 30 13:56:36.560690 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.560481    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr" event={"ID":"def5b5b9-fa1f-4d52-b32f-6ab32cbc552d","Type":"ContainerDied","Data":"9de2137453069efdc1c583cef0f480612916aff232ea836a038720a0649f4d90"}
Oct 30 13:56:36.572974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.572952    4993 scope.go:122] "RemoveContainer" containerID="447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe"
Oct 30 13:56:36.573430 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:56:36.573398    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe\": container with ID starting with 447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe not found: ID does not exist" containerID="447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe"
Oct 30 13:56:36.573561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.573442    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe"} err="failed to get container status \"447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe\": rpc error: code = NotFound desc = could not find container \"447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe\": container with ID starting with 447badd38840362915e6ad5aa2f9e901d2545d96d3c157d44d8a88cfceb2cdbe not found: ID does not exist"
Oct 30 13:56:36.573561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.573466    4993 scope.go:122] "RemoveContainer" containerID="7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf"
Oct 30 13:56:36.584974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.584947    4993 scope.go:122] "RemoveContainer" containerID="7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf"
Oct 30 13:56:36.585313 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:56:36.585293    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf\": container with ID starting with 7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf not found: ID does not exist" containerID="7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf"
Oct 30 13:56:36.585407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.585322    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf"} err="failed to get container status \"7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf\": rpc error: code = NotFound desc = could not find container \"7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf\": container with ID starting with 7839066decde20cfa446e267e3d81c186dc1e87e34ad5d1fe17b0586e10aafcf not found: ID does not exist"
Oct 30 13:56:36.611733 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.611626    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"]
Oct 30 13:56:36.615695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.615659    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t"]
Oct 30 13:56:36.652637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.652579    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"]
Oct 30 13:56:36.664876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.664837    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-2-w2-bb86fd-kserve-7dc66cb588-tnldr"]
Oct 30 13:56:36.733659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.733608    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/simulator-w2-bb86fd-kserve-556c6ddcc-kck9t" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:56:36.998304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998258    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"]
Oct 30 13:56:36.998859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998839    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main"
Oct 30 13:56:36.998954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998861    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main"
Oct 30 13:56:36.998954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998875    4993 cpu_manager.go:414] "containerMap: removing container" podUID="20c51228-21bd-4104-8fa8-eafb0954f18b" containerName="revoke-keys"
Oct 30 13:56:36.998954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998880    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c51228-21bd-4104-8fa8-eafb0954f18b" containerName="revoke-keys"
Oct 30 13:56:36.998954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998907    4993 cpu_manager.go:414] "containerMap: removing container" podUID="078c5fa3-22e6-4ef2-acc1-07fbc3b32fba" containerName="revoke-keys"
Oct 30 13:56:36.998954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998914    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078c5fa3-22e6-4ef2-acc1-07fbc3b32fba" containerName="revoke-keys"
Oct 30 13:56:36.998954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998922    4993 cpu_manager.go:414] "containerMap: removing container" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main"
Oct 30 13:56:36.998954 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.998927    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main"
Oct 30 13:56:36.999278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.999049    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" containerName="main"
Oct 30 13:56:36.999278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.999064    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c51228-21bd-4104-8fa8-eafb0954f18b" containerName="revoke-keys"
Oct 30 13:56:36.999278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.999081    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="078c5fa3-22e6-4ef2-acc1-07fbc3b32fba" containerName="revoke-keys"
Oct 30 13:56:36.999278 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:36.999091    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" containerName="main"
Oct 30 13:56:37.004484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.004462    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.019412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.019379    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:56:37.019412 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.019379    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:56:37.019610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.019480    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:56:37.019610 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.019533    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:56:37.027659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.027629    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"]
Oct 30 13:56:37.076784 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.075200    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.077950 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.077910    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jph\" (UniqueName: \"kubernetes.io/projected/78abaa13-96d4-4274-bc6c-cae18f53a4fe-kube-api-access-x8jph\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.078116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.078096    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.179764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.179723    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.179985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.179813    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jph\" (UniqueName: \"kubernetes.io/projected/78abaa13-96d4-4274-bc6c-cae18f53a4fe-kube-api-access-x8jph\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.179985 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.179910    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.180535 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.180507    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.182488 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.182454    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.203048 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.203011    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jph\" (UniqueName: \"kubernetes.io/projected/78abaa13-96d4-4274-bc6c-cae18f53a4fe-kube-api-access-x8jph\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.319797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.319697    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:37.624864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.624827    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"]
Oct 30 13:56:37.625867 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:56:37.625840    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78abaa13_96d4_4274_bc6c_cae18f53a4fe.slice/crio-7ee1cbea08818d90a545f4865a2f52b0f156c79a91f28b4db90eb77a2464a2ab WatchSource:0}: Error finding container 7ee1cbea08818d90a545f4865a2f52b0f156c79a91f28b4db90eb77a2464a2ab: Status 404 returned error can't find the container with id 7ee1cbea08818d90a545f4865a2f52b0f156c79a91f28b4db90eb77a2464a2ab
Oct 30 13:56:37.692042 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.692006    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9baa7e-a484-4d2e-aeb1-238028c1a77f" path="/var/lib/kubelet/pods/1f9baa7e-a484-4d2e-aeb1-238028c1a77f/volumes"
Oct 30 13:56:37.692483 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:37.692467    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def5b5b9-fa1f-4d52-b32f-6ab32cbc552d" path="/var/lib/kubelet/pods/def5b5b9-fa1f-4d52-b32f-6ab32cbc552d/volumes"
Oct 30 13:56:38.591399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:38.591351    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8" event={"ID":"78abaa13-96d4-4274-bc6c-cae18f53a4fe","Type":"ContainerStarted","Data":"5bac6b0292bcf8ca24fe63e361f1448990f2a33d35617dfd6d0f754f48c9b607"}
Oct 30 13:56:38.591777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:38.591739    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8" event={"ID":"78abaa13-96d4-4274-bc6c-cae18f53a4fe","Type":"ContainerStarted","Data":"7ee1cbea08818d90a545f4865a2f52b0f156c79a91f28b4db90eb77a2464a2ab"}
Oct 30 13:56:38.595358 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:56:38.595321    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:56:39.212841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.212807    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"]
Oct 30 13:56:39.225280 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.225241    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb86fd-9hq2s"]
Oct 30 13:56:39.600446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.600330    4993 generic.go:356] "Generic (PLEG): container finished" podID="78abaa13-96d4-4274-bc6c-cae18f53a4fe" containerID="5bac6b0292bcf8ca24fe63e361f1448990f2a33d35617dfd6d0f754f48c9b607" exitCode=0
Oct 30 13:56:39.600665 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.600642    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8" event={"ID":"78abaa13-96d4-4274-bc6c-cae18f53a4fe","Type":"ContainerDied","Data":"5bac6b0292bcf8ca24fe63e361f1448990f2a33d35617dfd6d0f754f48c9b607"}
Oct 30 13:56:39.694389 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.694348    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b138ed9a-40b7-4d83-ad58-90d8e20deddb" path="/var/lib/kubelet/pods/b138ed9a-40b7-4d83-ad58-90d8e20deddb/volumes"
Oct 30 13:56:39.770446 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.770409    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:39.852811 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.852669    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"]
Oct 30 13:56:39.857927 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.857885    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-241ffa-7997g"]
Oct 30 13:56:39.916135 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.916094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-openshift-ca\") pod \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") "
Oct 30 13:56:39.916323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.916157    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78abaa13-96d4-4274-bc6c-cae18f53a4fe-kube-api-access-x8jph\" (UniqueName: \"kubernetes.io/projected/78abaa13-96d4-4274-bc6c-cae18f53a4fe-kube-api-access-x8jph\") pod \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") "
Oct 30 13:56:39.916377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.916358    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-xks-ca\") pod \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\" (UID: \"78abaa13-96d4-4274-bc6c-cae18f53a4fe\") "
Oct 30 13:56:39.916591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.916564    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-openshift-ca" pod "78abaa13-96d4-4274-bc6c-cae18f53a4fe" (UID: "78abaa13-96d4-4274-bc6c-cae18f53a4fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:39.916849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.916824    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:39.918541 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.918510    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-xks-ca" pod "78abaa13-96d4-4274-bc6c-cae18f53a4fe" (UID: "78abaa13-96d4-4274-bc6c-cae18f53a4fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:39.918686 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:39.918564    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78abaa13-96d4-4274-bc6c-cae18f53a4fe-kube-api-access-x8jph" pod "78abaa13-96d4-4274-bc6c-cae18f53a4fe" (UID: "78abaa13-96d4-4274-bc6c-cae18f53a4fe"). InnerVolumeSpecName "kube-api-access-x8jph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:40.018829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:40.018773    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8jph\" (UniqueName: \"kubernetes.io/projected/78abaa13-96d4-4274-bc6c-cae18f53a4fe-kube-api-access-x8jph\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:40.018829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:40.018825    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78abaa13-96d4-4274-bc6c-cae18f53a4fe-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:40.610873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:40.610840    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"
Oct 30 13:56:40.610873 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:40.610852    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8" event={"ID":"78abaa13-96d4-4274-bc6c-cae18f53a4fe","Type":"ContainerDied","Data":"7ee1cbea08818d90a545f4865a2f52b0f156c79a91f28b4db90eb77a2464a2ab"}
Oct 30 13:56:40.611484 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:40.610903    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ee1cbea08818d90a545f4865a2f52b0f156c79a91f28b4db90eb77a2464a2ab"
Oct 30 13:56:41.621413 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.621378    4993 generic.go:356] "Generic (PLEG): container finished" podID="3315757c-d2b9-4025-adb4-1eb495aef420" containerID="42851da24d83b04cab294bcb1f4173bf18db947879ba216812821825de98934c" exitCode=0
Oct 30 13:56:41.621809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.621450    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" event={"ID":"3315757c-d2b9-4025-adb4-1eb495aef420","Type":"ContainerDied","Data":"42851da24d83b04cab294bcb1f4173bf18db947879ba216812821825de98934c"}
Oct 30 13:56:41.621809 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.621561    4993 scope.go:122] "RemoveContainer" containerID="175e46f7819ed55164201e0d734a1e0ccc3ce6eaea6334f08bc19aba60b84612"
Oct 30 13:56:41.679917 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.679834    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:56:41.692569 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.692535    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9945ecb8-daa4-4e7f-9593-9300220941f2" path="/var/lib/kubelet/pods/9945ecb8-daa4-4e7f-9593-9300220941f2/volumes"
Oct 30 13:56:41.841587 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841551    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3315757c-d2b9-4025-adb4-1eb495aef420-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3315757c-d2b9-4025-adb4-1eb495aef420-istiod-ca-cert\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.841804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841609    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-certs\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.841804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841630    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-envoy\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.841804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841673    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-kube-api-access-7ntp8\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-kube-api-access-7ntp8\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.841804 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841791    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3315757c-d2b9-4025-adb4-1eb495aef420-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3315757c-d2b9-4025-adb4-1eb495aef420-istio-podinfo\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.842053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841822    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-credential-socket\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.842053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841851    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-socket\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.842053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841886    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-istio-token\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-istio-token\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.842053 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.841919    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-data\") pod \"3315757c-d2b9-4025-adb4-1eb495aef420\" (UID: \"3315757c-d2b9-4025-adb4-1eb495aef420\") "
Oct 30 13:56:41.842357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.842149    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3315757c-d2b9-4025-adb4-1eb495aef420-istiod-ca-cert" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:41.842357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.842196    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-credential-socket" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:41.842479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.842430    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3315757c-d2b9-4025-adb4-1eb495aef420-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:41.842479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.842473    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:41.842586 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.842517    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-socket" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:41.842586 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.842572    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-certs" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:41.842772 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.842745    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-data" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:41.844258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.844231    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-envoy" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:41.844379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.844347    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-kube-api-access-7ntp8" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "kube-api-access-7ntp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:41.844551 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.844530    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3315757c-d2b9-4025-adb4-1eb495aef420-istio-podinfo" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:56:41.844622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.844605    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-istio-token" pod "3315757c-d2b9-4025-adb4-1eb495aef420" (UID: "3315757c-d2b9-4025-adb4-1eb495aef420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:41.943641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.943550    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3315757c-d2b9-4025-adb4-1eb495aef420-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:41.943641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.943583    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:41.943641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.943592    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:41.943641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.943603    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:41.943641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.943611    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:41.943641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.943618    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3315757c-d2b9-4025-adb4-1eb495aef420-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:41.943641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:41.943626    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ntp8\" (UniqueName: \"kubernetes.io/projected/3315757c-d2b9-4025-adb4-1eb495aef420-kube-api-access-7ntp8\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:42.630222 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:42.630183    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"
Oct 30 13:56:42.630753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:42.630216    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp" event={"ID":"3315757c-d2b9-4025-adb4-1eb495aef420","Type":"ContainerDied","Data":"525c1935cca9d867504d6ffe58d385d9a08f86573996557ee81e50822df37ab8"}
Oct 30 13:56:42.630753 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:42.630291    4993 scope.go:122] "RemoveContainer" containerID="42851da24d83b04cab294bcb1f4173bf18db947879ba216812821825de98934c"
Oct 30 13:56:42.726141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:42.726101    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"]
Oct 30 13:56:42.732122 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:42.732084    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-q97fp"]
Oct 30 13:56:43.693060 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:43.693019    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" path="/var/lib/kubelet/pods/3315757c-d2b9-4025-adb4-1eb495aef420/volumes"
Oct 30 13:56:47.493026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:47.492985    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"]
Oct 30 13:56:47.493562 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:47.493199    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" containerID="cri-o://600e5d2e1c2c3b236116ddc5d03781fdb73bc0c17c4999880f87af076b5bb734" gracePeriod=30
Oct 30 13:56:48.379579 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:48.379532    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:56:48.379773 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:48.379619    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:56:48.727338 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:48.727300    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"]
Oct 30 13:56:48.728138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:48.728106    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy" containerID="cri-o://63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85" gracePeriod=30
Oct 30 13:56:49.053978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.053854    4993 patch_prober.go:29] interesting pod/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:56:49.053978 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.053940    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:56:49.531748 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.531710    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"]
Oct 30 13:56:49.532171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.532158    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy"
Oct 30 13:56:49.532244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.532172    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy"
Oct 30 13:56:49.532244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.532217    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy"
Oct 30 13:56:49.532244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.532223    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy"
Oct 30 13:56:49.532244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.532230    4993 cpu_manager.go:414] "containerMap: removing container" podUID="78abaa13-96d4-4274-bc6c-cae18f53a4fe" containerName="revoke-keys"
Oct 30 13:56:49.532244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.532237    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78abaa13-96d4-4274-bc6c-cae18f53a4fe" containerName="revoke-keys"
Oct 30 13:56:49.532409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.532322    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy"
Oct 30 13:56:49.532409 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.532330    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="78abaa13-96d4-4274-bc6c-cae18f53a4fe" containerName="revoke-keys"
Oct 30 13:56:49.537178 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.537159    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.553759 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.553725    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:56:49.553970 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.553852    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:56:49.558722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.558692    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"]
Oct 30 13:56:49.628055 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.627998    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:56:49.628298 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.628246    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:56:49.647026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.646989    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfc2j\" (UniqueName: \"kubernetes.io/projected/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-kube-api-access-dfc2j\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.647026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.647028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.647244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.647196    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.748870 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.748834    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.749344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.748922    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfc2j\" (UniqueName: \"kubernetes.io/projected/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-kube-api-access-dfc2j\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.749344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.748948    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.749755 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.749731    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.751334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.751315    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.772979 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.772945    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfc2j\" (UniqueName: \"kubernetes.io/projected/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-kube-api-access-dfc2j\") pod \"maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:49.851485 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:49.851390    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:50.160642 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:50.160601    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"]
Oct 30 13:56:50.161264 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:56:50.161193    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d4efd8_e2f6_4bf9_b6a6_f0c9ea45eee2.slice/crio-8c3c1f97292341795d7dcb20c2bb6926c3c69d3380b995491c63f914f72b5d9a WatchSource:0}: Error finding container 8c3c1f97292341795d7dcb20c2bb6926c3c69d3380b995491c63f914f72b5d9a: Status 404 returned error can't find the container with id 8c3c1f97292341795d7dcb20c2bb6926c3c69d3380b995491c63f914f72b5d9a
Oct 30 13:56:50.705100 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:50.705060    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj" event={"ID":"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2","Type":"ContainerStarted","Data":"6aa0ffb0903d179435d3dca27ad287968f26a452f427a36cf9a51851fe170a30"}
Oct 30 13:56:50.705311 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:50.705114    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj" event={"ID":"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2","Type":"ContainerStarted","Data":"8c3c1f97292341795d7dcb20c2bb6926c3c69d3380b995491c63f914f72b5d9a"}
Oct 30 13:56:50.746463 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:50.746314    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj" podStartSLOduration=1.746294372 podStartE2EDuration="1.746294372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:50.744751759 +0000 UTC m=+5075.220471368" watchObservedRunningTime="2026-10-30 13:56:50.746294372 +0000 UTC m=+5075.222013984"
Oct 30 13:56:50.817578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:50.817530    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"]
Oct 30 13:56:50.821128 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:50.821095    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f49b40-jl9k8"]
Oct 30 13:56:51.691872 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:51.691827    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78abaa13-96d4-4274-bc6c-cae18f53a4fe" path="/var/lib/kubelet/pods/78abaa13-96d4-4274-bc6c-cae18f53a4fe/volumes"
Oct 30 13:56:51.713786 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:51.713746    4993 generic.go:356] "Generic (PLEG): container finished" podID="b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2" containerID="6aa0ffb0903d179435d3dca27ad287968f26a452f427a36cf9a51851fe170a30" exitCode=0
Oct 30 13:56:51.713963 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:51.713825    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj" event={"ID":"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2","Type":"ContainerDied","Data":"6aa0ffb0903d179435d3dca27ad287968f26a452f427a36cf9a51851fe170a30"}
Oct 30 13:56:52.727158 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.727117    4993 generic.go:356] "Generic (PLEG): container finished" podID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerID="600e5d2e1c2c3b236116ddc5d03781fdb73bc0c17c4999880f87af076b5bb734" exitCode=0
Oct 30 13:56:52.727650 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.727223    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" event={"ID":"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed","Type":"ContainerDied","Data":"600e5d2e1c2c3b236116ddc5d03781fdb73bc0c17c4999880f87af076b5bb734"}
Oct 30 13:56:52.786109 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.786080    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 13:56:52.877442 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.877413    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:52.891844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.891813    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-envoy\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.891877    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-certs\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.891939    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-kube-api-access-xvcwm\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892009 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.891989    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-podinfo\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892031    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-socket\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892061    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-credential-socket\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892174 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892150    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-xks-ca\") pod \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") "
Oct 30 13:56:52.892395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892181    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-token\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-token\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892276    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-kube-api-access-dfc2j\" (UniqueName: \"kubernetes.io/projected/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-kube-api-access-dfc2j\") pod \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") "
Oct 30 13:56:52.892395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892284    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-certs" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:52.892395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892326    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-openshift-ca\") pod \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\" (UID: \"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2\") "
Oct 30 13:56:52.892395 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892366    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istiod-ca-cert\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892397    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-data\") pod \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\" (UID: \"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed\") "
Oct 30 13:56:52.892661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892542    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-credential-socket" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:52.893090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892779    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-data" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:52.893090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892792    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-socket" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:52.893090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.892999    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.893090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.893022    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.893090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.893036    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.893090 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.893051    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.893529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.893123    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-openshift-ca" pod "b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2" (UID: "b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:52.893529 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.893388    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istiod-ca-cert" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:52.894777 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.894753    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-envoy" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:52.894877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.894799    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-podinfo" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:56:52.894877 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.894851    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-kube-api-access-xvcwm" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "kube-api-access-xvcwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:52.894971 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.894876    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-kube-api-access-dfc2j" pod "b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2" (UID: "b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2"). InnerVolumeSpecName "kube-api-access-dfc2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:52.895005 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.894960    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-xks-ca" pod "b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2" (UID: "b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:52.895190 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.895175    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-token" pod "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" (UID: "e500eab6-2355-4ae6-ba4f-a4181bf7e3ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:52.993687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.993648    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.993687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.993682    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.993687 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.993691    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfc2j\" (UniqueName: \"kubernetes.io/projected/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-kube-api-access-dfc2j\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.993907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.993703    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.993907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.993713    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.993907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.993721    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.993907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.993730    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvcwm\" (UniqueName: \"kubernetes.io/projected/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-kube-api-access-xvcwm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.993907 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:52.993738    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:53.740870 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:53.740836    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"
Oct 30 13:56:53.741437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:53.740873    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x" event={"ID":"e500eab6-2355-4ae6-ba4f-a4181bf7e3ed","Type":"ContainerDied","Data":"99e0ca9ac752f3d17262ecc9f1d7b32c677f629d168930a39a423046117ea963"}
Oct 30 13:56:53.741437 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:53.740949    4993 scope.go:122] "RemoveContainer" containerID="600e5d2e1c2c3b236116ddc5d03781fdb73bc0c17c4999880f87af076b5bb734"
Oct 30 13:56:53.742819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:53.742797    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"
Oct 30 13:56:53.742956 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:53.742921    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj" event={"ID":"b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2","Type":"ContainerDied","Data":"8c3c1f97292341795d7dcb20c2bb6926c3c69d3380b995491c63f914f72b5d9a"}
Oct 30 13:56:53.743021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:53.742966    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3c1f97292341795d7dcb20c2bb6926c3c69d3380b995491c63f914f72b5d9a"
Oct 30 13:56:53.824749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:53.824713    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"]
Oct 30 13:56:53.828080 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:53.828052    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-q5t8x"]
Oct 30 13:56:54.007290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.007265    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:56:54.104093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104005    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-kube-api-access-4dmv9\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-kube-api-access-4dmv9\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104054    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-socket\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104108    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-podinfo\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104147    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-token\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-token\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104173    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-data\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104231    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-envoy\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104348 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104276    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-credential-socket\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104365    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-certs\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104397    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12584623-fe58-48cb-b84c-2dc5012a7a7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12584623-fe58-48cb-b84c-2dc5012a7a7d-istiod-ca-cert\") pod \"12584623-fe58-48cb-b84c-2dc5012a7a7d\" (UID: \"12584623-fe58-48cb-b84c-2dc5012a7a7d\") "
Oct 30 13:56:54.104613 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104470    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-socket" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:54.104794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104667    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-data" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:54.104794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104778    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-certs" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:54.104982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104805    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12584623-fe58-48cb-b84c-2dc5012a7a7d-istiod-ca-cert" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:54.104982 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104913    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-credential-socket" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:54.105112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.104990    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.105112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.105010    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12584623-fe58-48cb-b84c-2dc5012a7a7d-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.105112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.105024    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.105112 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.105036    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.106547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.106520    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-podinfo" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:56:54.106654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.106606    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-token" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:54.106654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.106637    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-kube-api-access-4dmv9" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "kube-api-access-4dmv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:54.106737 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.106725    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-envoy" pod "12584623-fe58-48cb-b84c-2dc5012a7a7d" (UID: "12584623-fe58-48cb-b84c-2dc5012a7a7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:54.206621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.206583    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.206621 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.206622    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.206853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.206637    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.206853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.206651    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12584623-fe58-48cb-b84c-2dc5012a7a7d-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.206853 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.206666    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dmv9\" (UniqueName: \"kubernetes.io/projected/12584623-fe58-48cb-b84c-2dc5012a7a7d-kube-api-access-4dmv9\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:56:54.246124 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.246084    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"]
Oct 30 13:56:54.251098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.251057    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"]
Oct 30 13:56:54.261899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.261860    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"]
Oct 30 13:56:54.272886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.272850    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"]
Oct 30 13:56:54.752632 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.752594    4993 generic.go:356] "Generic (PLEG): container finished" podID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerID="63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85" exitCode=0
Oct 30 13:56:54.753194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.752670    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"
Oct 30 13:56:54.753194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.752675    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" event={"ID":"12584623-fe58-48cb-b84c-2dc5012a7a7d","Type":"ContainerDied","Data":"63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85"}
Oct 30 13:56:54.753194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.752764    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2" event={"ID":"12584623-fe58-48cb-b84c-2dc5012a7a7d","Type":"ContainerDied","Data":"8bcfbf1f598a68b90eab2aa83459e291f2c4b471d69ba0b083d64aa93c088f24"}
Oct 30 13:56:54.753194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.752797    4993 scope.go:122] "RemoveContainer" containerID="63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85"
Oct 30 13:56:54.766531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.766505    4993 scope.go:122] "RemoveContainer" containerID="bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72"
Oct 30 13:56:54.779460 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.779435    4993 scope.go:122] "RemoveContainer" containerID="63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85"
Oct 30 13:56:54.779786 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:56:54.779758    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85\": container with ID starting with 63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85 not found: ID does not exist" containerID="63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85"
Oct 30 13:56:54.779841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.779797    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85"} err="failed to get container status \"63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85\": rpc error: code = NotFound desc = could not find container \"63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85\": container with ID starting with 63c5df0df1dbd1408dde752bcc35cfa69b313024652bbad5985aaee5ac46dd85 not found: ID does not exist"
Oct 30 13:56:54.779841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.779815    4993 scope.go:122] "RemoveContainer" containerID="bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72"
Oct 30 13:56:54.780093 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:56:54.780073    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72\": container with ID starting with bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72 not found: ID does not exist" containerID="bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72"
Oct 30 13:56:54.780157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.780104    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72"} err="failed to get container status \"bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72\": rpc error: code = NotFound desc = could not find container \"bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72\": container with ID starting with bdfe9ab35460dfaf2e7888bdf8b99b18d6d12385b67adcb1b720e26f18ab8f72 not found: ID does not exist"
Oct 30 13:56:54.806145 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.806110    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"]
Oct 30 13:56:54.812360 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:54.812334    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-241ffa-openshift-default-79cfbc884f-q98l2"]
Oct 30 13:56:55.691346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:55.691310    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" path="/var/lib/kubelet/pods/12584623-fe58-48cb-b84c-2dc5012a7a7d/volumes"
Oct 30 13:56:55.691909 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:56:55.691885    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" path="/var/lib/kubelet/pods/e500eab6-2355-4ae6-ba4f-a4181bf7e3ed/volumes"
Oct 30 13:57:00.028244 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028124    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"]
Oct 30 13:57:00.028662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028630    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy"
Oct 30 13:57:00.028662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028644    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy"
Oct 30 13:57:00.028662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028657    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2" containerName="revoke-keys"
Oct 30 13:57:00.028662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028662    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2" containerName="revoke-keys"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028675    4993 cpu_manager.go:414] "containerMap: removing container" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028685    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028695    4993 cpu_manager.go:414] "containerMap: removing container" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028700    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028776    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2" containerName="revoke-keys"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028784    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028794    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="12584623-fe58-48cb-b84c-2dc5012a7a7d" containerName="istio-proxy"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028802    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3315757c-d2b9-4025-adb4-1eb495aef420" containerName="istio-proxy"
Oct 30 13:57:00.028819 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.028812    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e500eab6-2355-4ae6-ba4f-a4181bf7e3ed" containerName="istio-proxy"
Oct 30 13:57:00.032142 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.032114    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.056323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.056287    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"]
Oct 30 13:57:00.068073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.068038    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:57:00.068073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.068042    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:57:00.090505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.090459    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7m55\" (UniqueName: \"kubernetes.io/projected/5e0174df-681a-4867-834c-f790afda4824-kube-api-access-d7m55\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.090722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.090629    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0174df-681a-4867-834c-f790afda4824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.090722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.090686    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0174df-681a-4867-834c-f790afda4824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.112186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.112137    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:57:00.112186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.112173    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:57:00.191992 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.191950    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7m55\" (UniqueName: \"kubernetes.io/projected/5e0174df-681a-4867-834c-f790afda4824-kube-api-access-d7m55\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.192253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.192021    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0174df-681a-4867-834c-f790afda4824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.192253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.192058    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0174df-681a-4867-834c-f790afda4824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.192744 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.192716    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0174df-681a-4867-834c-f790afda4824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.194714 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.194688    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0174df-681a-4867-834c-f790afda4824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.215554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.215518    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7m55\" (UniqueName: \"kubernetes.io/projected/5e0174df-681a-4867-834c-f790afda4824-kube-api-access-d7m55\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.346727 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.346631    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:00.657563 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:57:00.657470    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0174df_681a_4867_834c_f790afda4824.slice/crio-ad3aac18c4c826c55b03b663e97173e9d0c92f6acddf27d9d36b25a4b56283d1 WatchSource:0}: Error finding container ad3aac18c4c826c55b03b663e97173e9d0c92f6acddf27d9d36b25a4b56283d1: Status 404 returned error can't find the container with id ad3aac18c4c826c55b03b663e97173e9d0c92f6acddf27d9d36b25a4b56283d1
Oct 30 13:57:00.661662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.661630    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"]
Oct 30 13:57:00.810064 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:00.810024    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk" event={"ID":"5e0174df-681a-4867-834c-f790afda4824","Type":"ContainerStarted","Data":"ad3aac18c4c826c55b03b663e97173e9d0c92f6acddf27d9d36b25a4b56283d1"}
Oct 30 13:57:01.057726 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.057170    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"]
Oct 30 13:57:01.062598 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.061652    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.088637 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.088598    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"]
Oct 30 13:57:01.205494 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.205454    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctlp\" (UniqueName: \"kubernetes.io/projected/b3533639-e576-418f-8c58-2e538f08ae0c-kube-api-access-gctlp\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.205699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.205611    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.205699 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.205650    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.306519 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.306471    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctlp\" (UniqueName: \"kubernetes.io/projected/b3533639-e576-418f-8c58-2e538f08ae0c-kube-api-access-gctlp\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.306736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.306600    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.306736 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.306634    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.307507 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.307476    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.309262 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.309237    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.329662 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.329628    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctlp\" (UniqueName: \"kubernetes.io/projected/b3533639-e576-418f-8c58-2e538f08ae0c-kube-api-access-gctlp\") pod \"maas-api-revoke-sub-e16b61b5-7pzkh\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.404889 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.404848    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:01.738747 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.738716    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"]
Oct 30 13:57:01.740450 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:57:01.740384    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3533639_e576_418f_8c58_2e538f08ae0c.slice/crio-f3d61bdaccb2400c572be8e1ec5712e37c507405ccfae7354deaf240fb20aa31 WatchSource:0}: Error finding container f3d61bdaccb2400c572be8e1ec5712e37c507405ccfae7354deaf240fb20aa31: Status 404 returned error can't find the container with id f3d61bdaccb2400c572be8e1ec5712e37c507405ccfae7354deaf240fb20aa31
Oct 30 13:57:01.819923 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.819885    4993 generic.go:356] "Generic (PLEG): container finished" podID="5e0174df-681a-4867-834c-f790afda4824" containerID="3d7b762872a50702444abac0519c4caa60f8de704e9997df53adbb98d1d62329" exitCode=0
Oct 30 13:57:01.820082 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.819959    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk" event={"ID":"5e0174df-681a-4867-834c-f790afda4824","Type":"ContainerDied","Data":"3d7b762872a50702444abac0519c4caa60f8de704e9997df53adbb98d1d62329"}
Oct 30 13:57:01.821557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:01.821525    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh" event={"ID":"b3533639-e576-418f-8c58-2e538f08ae0c","Type":"ContainerStarted","Data":"f3d61bdaccb2400c572be8e1ec5712e37c507405ccfae7354deaf240fb20aa31"}
Oct 30 13:57:02.832015 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:02.831926    4993 generic.go:356] "Generic (PLEG): container finished" podID="b3533639-e576-418f-8c58-2e538f08ae0c" containerID="72bb41f29310dbe5a84b80cbe21eccc56bab17b350b1825c74ed25b90288a135" exitCode=0
Oct 30 13:57:02.832462 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:02.832007    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh" event={"ID":"b3533639-e576-418f-8c58-2e538f08ae0c","Type":"ContainerDied","Data":"72bb41f29310dbe5a84b80cbe21eccc56bab17b350b1825c74ed25b90288a135"}
Oct 30 13:57:02.994418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:02.994385    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:03.125822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.125713    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e0174df-681a-4867-834c-f790afda4824-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0174df-681a-4867-834c-f790afda4824-maas-api-xks-ca\") pod \"5e0174df-681a-4867-834c-f790afda4824\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") "
Oct 30 13:57:03.125822 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.125794    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0174df-681a-4867-834c-f790afda4824-kube-api-access-d7m55\" (UniqueName: \"kubernetes.io/projected/5e0174df-681a-4867-834c-f790afda4824-kube-api-access-d7m55\") pod \"5e0174df-681a-4867-834c-f790afda4824\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") "
Oct 30 13:57:03.126094 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.125824    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0174df-681a-4867-834c-f790afda4824-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0174df-681a-4867-834c-f790afda4824-maas-api-openshift-ca\") pod \"5e0174df-681a-4867-834c-f790afda4824\" (UID: \"5e0174df-681a-4867-834c-f790afda4824\") "
Oct 30 13:57:03.126258 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.126231    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0174df-681a-4867-834c-f790afda4824-maas-api-openshift-ca" pod "5e0174df-681a-4867-834c-f790afda4824" (UID: "5e0174df-681a-4867-834c-f790afda4824"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:03.128163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.128130    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0174df-681a-4867-834c-f790afda4824-kube-api-access-d7m55" pod "5e0174df-681a-4867-834c-f790afda4824" (UID: "5e0174df-681a-4867-834c-f790afda4824"). InnerVolumeSpecName "kube-api-access-d7m55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:03.128309 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.128194    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0174df-681a-4867-834c-f790afda4824-maas-api-xks-ca" pod "5e0174df-681a-4867-834c-f790afda4824" (UID: "5e0174df-681a-4867-834c-f790afda4824"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:03.227844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.227803    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0174df-681a-4867-834c-f790afda4824-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:03.227844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.227838    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7m55\" (UniqueName: \"kubernetes.io/projected/5e0174df-681a-4867-834c-f790afda4824-kube-api-access-d7m55\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:03.227844 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.227849    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0174df-681a-4867-834c-f790afda4824-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:03.840590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.840557    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"
Oct 30 13:57:03.841030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.840585    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk" event={"ID":"5e0174df-681a-4867-834c-f790afda4824","Type":"ContainerDied","Data":"ad3aac18c4c826c55b03b663e97173e9d0c92f6acddf27d9d36b25a4b56283d1"}
Oct 30 13:57:03.841030 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.840640    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad3aac18c4c826c55b03b663e97173e9d0c92f6acddf27d9d36b25a4b56283d1"
Oct 30 13:57:03.994862 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:03.994834    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:04.137393 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.137353    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-openshift-ca\") pod \"b3533639-e576-418f-8c58-2e538f08ae0c\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") "
Oct 30 13:57:04.137597 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.137553    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3533639-e576-418f-8c58-2e538f08ae0c-kube-api-access-gctlp\" (UniqueName: \"kubernetes.io/projected/b3533639-e576-418f-8c58-2e538f08ae0c-kube-api-access-gctlp\") pod \"b3533639-e576-418f-8c58-2e538f08ae0c\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") "
Oct 30 13:57:04.137644 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.137599    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-xks-ca\") pod \"b3533639-e576-418f-8c58-2e538f08ae0c\" (UID: \"b3533639-e576-418f-8c58-2e538f08ae0c\") "
Oct 30 13:57:04.137849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.137828    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-openshift-ca" pod "b3533639-e576-418f-8c58-2e538f08ae0c" (UID: "b3533639-e576-418f-8c58-2e538f08ae0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:04.137976 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.137950    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.139818 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.139795    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3533639-e576-418f-8c58-2e538f08ae0c-kube-api-access-gctlp" pod "b3533639-e576-418f-8c58-2e538f08ae0c" (UID: "b3533639-e576-418f-8c58-2e538f08ae0c"). InnerVolumeSpecName "kube-api-access-gctlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:04.139928 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.139873    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-xks-ca" pod "b3533639-e576-418f-8c58-2e538f08ae0c" (UID: "b3533639-e576-418f-8c58-2e538f08ae0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:04.239424 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.239382    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctlp\" (UniqueName: \"kubernetes.io/projected/b3533639-e576-418f-8c58-2e538f08ae0c-kube-api-access-gctlp\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.239424 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.239425    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3533639-e576-418f-8c58-2e538f08ae0c-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.860910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.860868    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh" event={"ID":"b3533639-e576-418f-8c58-2e538f08ae0c","Type":"ContainerDied","Data":"f3d61bdaccb2400c572be8e1ec5712e37c507405ccfae7354deaf240fb20aa31"}
Oct 30 13:57:04.861479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.860950    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3d61bdaccb2400c572be8e1ec5712e37c507405ccfae7354deaf240fb20aa31"
Oct 30 13:57:04.861479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:04.861000    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b61b5-7pzkh"
Oct 30 13:57:08.570941 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:57:08.570901    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:57:09.268659 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.268607    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main" containerID="cri-o://df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876" gracePeriod=45
Oct 30 13:57:09.273959 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.273926    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main" containerID="cri-o://86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d" gracePeriod=45
Oct 30 13:57:09.286274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.286188    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main" containerID="cri-o://015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c" gracePeriod=45
Oct 30 13:57:09.295167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.295123    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main" containerID="cri-o://a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052" gracePeriod=45
Oct 30 13:57:09.575333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.575110    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 13:57:09.596724 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.596679    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 13:57:09.625735 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.625699    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 13:57:09.653745 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.653714    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 13:57:09.711883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.711843    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-tmp-dir\") pod \"9c48a685-e866-47d9-8673-673bb5bd0be3\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") "
Oct 30 13:57:09.712059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.711904    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-home\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-home\") pod \"9c48a685-e866-47d9-8673-673bb5bd0be3\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") "
Oct 30 13:57:09.712059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.711938    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c48a685-e866-47d9-8673-673bb5bd0be3-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c48a685-e866-47d9-8673-673bb5bd0be3-tls-certs\") pod \"9c48a685-e866-47d9-8673-673bb5bd0be3\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") "
Oct 30 13:57:09.712059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.711966    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-model-cache\") pod \"f7b10838-9748-48ad-af2f-71502ed36435\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") "
Oct 30 13:57:09.712059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712008    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-home\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-home\") pod \"0769aaca-8e70-4393-a400-5519741bc5e6\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") "
Oct 30 13:57:09.712059 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712048    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-home\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-home\") pod \"f7b10838-9748-48ad-af2f-71502ed36435\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") "
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712094    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b10838-9748-48ad-af2f-71502ed36435-kube-api-access-jlpq2\" (UniqueName: \"kubernetes.io/projected/f7b10838-9748-48ad-af2f-71502ed36435-kube-api-access-jlpq2\") pod \"f7b10838-9748-48ad-af2f-71502ed36435\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") "
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712122    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-model-cache\") pod \"0769aaca-8e70-4393-a400-5519741bc5e6\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") "
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712167    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c48a685-e866-47d9-8673-673bb5bd0be3-kube-api-access-p8fbm\" (UniqueName: \"kubernetes.io/projected/9c48a685-e866-47d9-8673-673bb5bd0be3-kube-api-access-p8fbm\") pod \"9c48a685-e866-47d9-8673-673bb5bd0be3\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") "
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712237    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-dshm\") pod \"9c48a685-e866-47d9-8673-673bb5bd0be3\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") "
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712268    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-dshm\") pod \"0769aaca-8e70-4393-a400-5519741bc5e6\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") "
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712277    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-home" pod "9c48a685-e866-47d9-8673-673bb5bd0be3" (UID: "9c48a685-e866-47d9-8673-673bb5bd0be3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712319    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-model-cache\") pod \"9c48a685-e866-47d9-8673-673bb5bd0be3\" (UID: \"9c48a685-e866-47d9-8673-673bb5bd0be3\") "
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712358    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-tmp-dir\") pod \"0769aaca-8e70-4393-a400-5519741bc5e6\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") "
Oct 30 13:57:09.712418 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712394    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0769aaca-8e70-4393-a400-5519741bc5e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0769aaca-8e70-4393-a400-5519741bc5e6-tls-certs\") pod \"0769aaca-8e70-4393-a400-5519741bc5e6\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") "
Oct 30 13:57:09.713062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712448    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-dshm\") pod \"f7b10838-9748-48ad-af2f-71502ed36435\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") "
Oct 30 13:57:09.713062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712480    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b10838-9748-48ad-af2f-71502ed36435-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b10838-9748-48ad-af2f-71502ed36435-tls-certs\") pod \"f7b10838-9748-48ad-af2f-71502ed36435\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") "
Oct 30 13:57:09.713062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712634    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-home" pod "0769aaca-8e70-4393-a400-5519741bc5e6" (UID: "0769aaca-8e70-4393-a400-5519741bc5e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.713062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712782    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-tmp-dir" pod "9c48a685-e866-47d9-8673-673bb5bd0be3" (UID: "9c48a685-e866-47d9-8673-673bb5bd0be3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.713062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712847    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-model-cache" pod "f7b10838-9748-48ad-af2f-71502ed36435" (UID: "f7b10838-9748-48ad-af2f-71502ed36435"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.713062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.712946    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-home" pod "f7b10838-9748-48ad-af2f-71502ed36435" (UID: "f7b10838-9748-48ad-af2f-71502ed36435"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.713399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.713193    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0769aaca-8e70-4393-a400-5519741bc5e6-kube-api-access-b7m7g\" (UniqueName: \"kubernetes.io/projected/0769aaca-8e70-4393-a400-5519741bc5e6-kube-api-access-b7m7g\") pod \"0769aaca-8e70-4393-a400-5519741bc5e6\" (UID: \"0769aaca-8e70-4393-a400-5519741bc5e6\") "
Oct 30 13:57:09.713399 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.713308    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-tmp-dir\") pod \"f7b10838-9748-48ad-af2f-71502ed36435\" (UID: \"f7b10838-9748-48ad-af2f-71502ed36435\") "
Oct 30 13:57:09.713849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.713823    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-tmp-dir" pod "0769aaca-8e70-4393-a400-5519741bc5e6" (UID: "0769aaca-8e70-4393-a400-5519741bc5e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.714315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.714288    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-model-cache" pod "9c48a685-e866-47d9-8673-673bb5bd0be3" (UID: "9c48a685-e866-47d9-8673-673bb5bd0be3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.714713 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.714688    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.714801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.714719    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.714801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.714734    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.714801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.714746    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.714801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.714758    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.714801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.714770    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.714801 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.714783    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.715147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.715029    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-tmp-dir" pod "f7b10838-9748-48ad-af2f-71502ed36435" (UID: "f7b10838-9748-48ad-af2f-71502ed36435"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.715302 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.715280    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-model-cache" pod "0769aaca-8e70-4393-a400-5519741bc5e6" (UID: "0769aaca-8e70-4393-a400-5519741bc5e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.715591 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.715563    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-dshm" pod "0769aaca-8e70-4393-a400-5519741bc5e6" (UID: "0769aaca-8e70-4393-a400-5519741bc5e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.715783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.715755    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b10838-9748-48ad-af2f-71502ed36435-tls-certs" pod "f7b10838-9748-48ad-af2f-71502ed36435" (UID: "f7b10838-9748-48ad-af2f-71502ed36435"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:09.716930 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.716874    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0769aaca-8e70-4393-a400-5519741bc5e6-kube-api-access-b7m7g" pod "0769aaca-8e70-4393-a400-5519741bc5e6" (UID: "0769aaca-8e70-4393-a400-5519741bc5e6"). InnerVolumeSpecName "kube-api-access-b7m7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:09.717657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.717532    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b10838-9748-48ad-af2f-71502ed36435-kube-api-access-jlpq2" pod "f7b10838-9748-48ad-af2f-71502ed36435" (UID: "f7b10838-9748-48ad-af2f-71502ed36435"). InnerVolumeSpecName "kube-api-access-jlpq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:09.717657 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.717626    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c48a685-e866-47d9-8673-673bb5bd0be3-kube-api-access-p8fbm" pod "9c48a685-e866-47d9-8673-673bb5bd0be3" (UID: "9c48a685-e866-47d9-8673-673bb5bd0be3"). InnerVolumeSpecName "kube-api-access-p8fbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:09.717924 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.717880    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c48a685-e866-47d9-8673-673bb5bd0be3-tls-certs" pod "9c48a685-e866-47d9-8673-673bb5bd0be3" (UID: "9c48a685-e866-47d9-8673-673bb5bd0be3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:09.718940 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.718909    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0769aaca-8e70-4393-a400-5519741bc5e6-tls-certs" pod "0769aaca-8e70-4393-a400-5519741bc5e6" (UID: "0769aaca-8e70-4393-a400-5519741bc5e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:09.719426 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.719395    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-dshm" pod "9c48a685-e866-47d9-8673-673bb5bd0be3" (UID: "9c48a685-e866-47d9-8673-673bb5bd0be3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.719667 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.719647    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-dshm" pod "f7b10838-9748-48ad-af2f-71502ed36435" (UID: "f7b10838-9748-48ad-af2f-71502ed36435"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.816037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.815937    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tls-certs\") pod \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") "
Oct 30 13:57:09.816037 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.816002    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efd7f65-f9e5-489c-b010-1ce9b04311eb-kube-api-access-bpj4b\" (UniqueName: \"kubernetes.io/projected/7efd7f65-f9e5-489c-b010-1ce9b04311eb-kube-api-access-bpj4b\") pod \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") "
Oct 30 13:57:09.816702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.816047    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tmp-dir\") pod \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") "
Oct 30 13:57:09.816702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.816120    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-home\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-home\") pod \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") "
Oct 30 13:57:09.816702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.816156    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-model-cache\") pod \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") "
Oct 30 13:57:09.816702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.816187    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-dshm\") pod \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\" (UID: \"7efd7f65-f9e5-489c-b010-1ce9b04311eb\") "
Oct 30 13:57:09.816702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.816395    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tmp-dir" pod "7efd7f65-f9e5-489c-b010-1ce9b04311eb" (UID: "7efd7f65-f9e5-489c-b010-1ce9b04311eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.816702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.816447    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-home" pod "7efd7f65-f9e5-489c-b010-1ce9b04311eb" (UID: "7efd7f65-f9e5-489c-b010-1ce9b04311eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.816702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.816534    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-model-cache" pod "7efd7f65-f9e5-489c-b010-1ce9b04311eb" (UID: "7efd7f65-f9e5-489c-b010-1ce9b04311eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.817160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817070    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c48a685-e866-47d9-8673-673bb5bd0be3-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817099    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817114    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817125    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlpq2\" (UniqueName: \"kubernetes.io/projected/f7b10838-9748-48ad-af2f-71502ed36435-kube-api-access-jlpq2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817138    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817153    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8fbm\" (UniqueName: \"kubernetes.io/projected/9c48a685-e866-47d9-8673-673bb5bd0be3-kube-api-access-p8fbm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817160 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817164    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c48a685-e866-47d9-8673-673bb5bd0be3-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817173    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0769aaca-8e70-4393-a400-5519741bc5e6-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817191    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0769aaca-8e70-4393-a400-5519741bc5e6-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817200    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817237    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b10838-9748-48ad-af2f-71502ed36435-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817250    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7m7g\" (UniqueName: \"kubernetes.io/projected/0769aaca-8e70-4393-a400-5519741bc5e6-kube-api-access-b7m7g\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817262    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.817573 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.817275    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b10838-9748-48ad-af2f-71502ed36435-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.818577 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.818547    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efd7f65-f9e5-489c-b010-1ce9b04311eb-kube-api-access-bpj4b" pod "7efd7f65-f9e5-489c-b010-1ce9b04311eb" (UID: "7efd7f65-f9e5-489c-b010-1ce9b04311eb"). InnerVolumeSpecName "kube-api-access-bpj4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:09.818869 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.818846    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tls-certs" pod "7efd7f65-f9e5-489c-b010-1ce9b04311eb" (UID: "7efd7f65-f9e5-489c-b010-1ce9b04311eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:09.819105 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.819081    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-dshm" pod "7efd7f65-f9e5-489c-b010-1ce9b04311eb" (UID: "7efd7f65-f9e5-489c-b010-1ce9b04311eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:09.903085 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.903047    4993 generic.go:356] "Generic (PLEG): container finished" podID="0769aaca-8e70-4393-a400-5519741bc5e6" containerID="015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c" exitCode=0
Oct 30 13:57:09.903329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.903095    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"
Oct 30 13:57:09.903329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.903129    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" event={"ID":"0769aaca-8e70-4393-a400-5519741bc5e6","Type":"ContainerDied","Data":"015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c"}
Oct 30 13:57:09.903329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.903178    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw" event={"ID":"0769aaca-8e70-4393-a400-5519741bc5e6","Type":"ContainerDied","Data":"622ff6208a3f2681cd87bfd60ef41e966cf90d9e0197613201e2e944e88e36fe"}
Oct 30 13:57:09.903329 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.903200    4993 scope.go:122] "RemoveContainer" containerID="015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c"
Oct 30 13:57:09.905678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.905647    4993 generic.go:356] "Generic (PLEG): container finished" podID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerID="a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052" exitCode=0
Oct 30 13:57:09.905808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.905685    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"
Oct 30 13:57:09.905808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.905716    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" event={"ID":"7efd7f65-f9e5-489c-b010-1ce9b04311eb","Type":"ContainerDied","Data":"a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052"}
Oct 30 13:57:09.905808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.905761    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph" event={"ID":"7efd7f65-f9e5-489c-b010-1ce9b04311eb","Type":"ContainerDied","Data":"dcd53e179fcf04545b1fb8a8d120359816ab441b5c854103d713453f075f5beb"}
Oct 30 13:57:09.907561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.907528    4993 generic.go:356] "Generic (PLEG): container finished" podID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerID="df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876" exitCode=0
Oct 30 13:57:09.907692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.907562    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" event={"ID":"9c48a685-e866-47d9-8673-673bb5bd0be3","Type":"ContainerDied","Data":"df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876"}
Oct 30 13:57:09.907692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.907580    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"
Oct 30 13:57:09.907692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.907602    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s" event={"ID":"9c48a685-e866-47d9-8673-673bb5bd0be3","Type":"ContainerDied","Data":"1842e3e50d7a2e45321474bef409216fe989e25c9a70a8d219c6a35183d27d91"}
Oct 30 13:57:09.909315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.909291    4993 generic.go:356] "Generic (PLEG): container finished" podID="f7b10838-9748-48ad-af2f-71502ed36435" containerID="86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d" exitCode=0
Oct 30 13:57:09.909443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.909330    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" event={"ID":"f7b10838-9748-48ad-af2f-71502ed36435","Type":"ContainerDied","Data":"86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d"}
Oct 30 13:57:09.909443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.909380    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4" event={"ID":"f7b10838-9748-48ad-af2f-71502ed36435","Type":"ContainerDied","Data":"22b1eb535df7a99e5897a6da45e729505f4d01c425a025abd96f15c0647b4cd2"}
Oct 30 13:57:09.909443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.909385    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"
Oct 30 13:57:09.918631 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.918600    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7efd7f65-f9e5-489c-b010-1ce9b04311eb-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.918763 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.918641    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7efd7f65-f9e5-489c-b010-1ce9b04311eb-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.918763 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.918656    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpj4b\" (UniqueName: \"kubernetes.io/projected/7efd7f65-f9e5-489c-b010-1ce9b04311eb-kube-api-access-bpj4b\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:09.923390 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.923099    4993 scope.go:122] "RemoveContainer" containerID="015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c"
Oct 30 13:57:09.923788 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:57:09.923757    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c\": container with ID starting with 015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c not found: ID does not exist" containerID="015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c"
Oct 30 13:57:09.923903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.923799    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c"} err="failed to get container status \"015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c\": rpc error: code = NotFound desc = could not find container \"015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c\": container with ID starting with 015c944d1935443761bea3adc909ed20f1069f3f926c6ff2dd6d0219e97a5e8c not found: ID does not exist"
Oct 30 13:57:09.923903 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.923823    4993 scope.go:122] "RemoveContainer" containerID="a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052"
Oct 30 13:57:09.938774 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.938750    4993 scope.go:122] "RemoveContainer" containerID="a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052"
Oct 30 13:57:09.939073 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:57:09.939045    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052\": container with ID starting with a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052 not found: ID does not exist" containerID="a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052"
Oct 30 13:57:09.939185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.939082    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052"} err="failed to get container status \"a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052\": rpc error: code = NotFound desc = could not find container \"a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052\": container with ID starting with a8008509ea6145a16c10382696eab01e775d48e138fde4f3cfe8fe46abae9052 not found: ID does not exist"
Oct 30 13:57:09.939185 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.939103    4993 scope.go:122] "RemoveContainer" containerID="df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876"
Oct 30 13:57:09.950018 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.950000    4993 scope.go:122] "RemoveContainer" containerID="df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876"
Oct 30 13:57:09.950374 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:57:09.950350    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876\": container with ID starting with df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876 not found: ID does not exist" containerID="df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876"
Oct 30 13:57:09.950491 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.950386    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876"} err="failed to get container status \"df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876\": rpc error: code = NotFound desc = could not find container \"df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876\": container with ID starting with df832548fc032e5fed9319c2c19247c03cac035a9404aaff7ab58dd8721e6876 not found: ID does not exist"
Oct 30 13:57:09.950491 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.950410    4993 scope.go:122] "RemoveContainer" containerID="86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d"
Oct 30 13:57:09.976682 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.976643    4993 scope.go:122] "RemoveContainer" containerID="86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d"
Oct 30 13:57:09.977274 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:57:09.977132    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d\": container with ID starting with 86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d not found: ID does not exist" containerID="86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d"
Oct 30 13:57:09.977274 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:09.977179    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d"} err="failed to get container status \"86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d\": rpc error: code = NotFound desc = could not find container \"86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d\": container with ID starting with 86c09f780bb9fc8a2815a250982be47f8990fa53af2bef8e2e7c946b1640fd0d not found: ID does not exist"
Oct 30 13:57:10.005715 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.005677    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"]
Oct 30 13:57:10.009429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.009397    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/premium-w0-7a8b03-kserve-69c789c955-4q7nw"]
Oct 30 13:57:10.030325 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.030291    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"]
Oct 30 13:57:10.034912 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.034884    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad16cd7-jp9t2"]
Oct 30 13:57:10.040230 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.040182    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"]
Oct 30 13:57:10.044283 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.044259    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/simulator-w0-7a8b03-kserve-5dc64556f4-fqpph"]
Oct 30 13:57:10.077093 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.077017    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"]
Oct 30 13:57:10.080693 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.080666    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-w0-7a8b03-kserve-759c498594-2lxw4"]
Oct 30 13:57:10.115505 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.115475    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"]
Oct 30 13:57:10.118557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:10.118530    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7a8b03/distinct-2-w0-7a8b03-kserve-56d8776969-nr45s"]
Oct 30 13:57:11.690515 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:11.690479    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" path="/var/lib/kubelet/pods/0769aaca-8e70-4393-a400-5519741bc5e6/volumes"
Oct 30 13:57:11.690937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:11.690926    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cfa9ee8-bb60-4c72-af4a-8c4e48084bac" path="/var/lib/kubelet/pods/2cfa9ee8-bb60-4c72-af4a-8c4e48084bac/volumes"
Oct 30 13:57:11.691330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:11.691316    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" path="/var/lib/kubelet/pods/7efd7f65-f9e5-489c-b010-1ce9b04311eb/volumes"
Oct 30 13:57:11.692022 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:11.692006    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" path="/var/lib/kubelet/pods/9c48a685-e866-47d9-8673-673bb5bd0be3/volumes"
Oct 30 13:57:11.692429 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:11.692415    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b10838-9748-48ad-af2f-71502ed36435" path="/var/lib/kubelet/pods/f7b10838-9748-48ad-af2f-71502ed36435/volumes"
Oct 30 13:57:13.752180 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:13.752146    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"]
Oct 30 13:57:13.763585 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:13.763549    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-451a21-jw6vk"]
Oct 30 13:57:15.690588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:15.690551    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0174df-681a-4867-834c-f790afda4824" path="/var/lib/kubelet/pods/5e0174df-681a-4867-834c-f790afda4824/volumes"
Oct 30 13:57:15.899989 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:15.899953    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"]
Oct 30 13:57:15.911153 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:15.911116    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7a8b03-92bjj"]
Oct 30 13:57:16.654137 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:16.654110    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:57:16.703088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:16.703057    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:57:17.691782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:17.691736    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2" path="/var/lib/kubelet/pods/b4d4efd8-e2f6-4bf9-b6a6-f0c9ea45eee2/volumes"
Oct 30 13:57:20.037317 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:20.037273    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"]
Oct 30 13:57:20.064749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:20.056023    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7e4664-jfndt"]
Oct 30 13:57:21.693156 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:21.693119    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a583310-a797-4086-b44d-a085a01abd9a" path="/var/lib/kubelet/pods/0a583310-a797-4086-b44d-a085a01abd9a/volumes"
Oct 30 13:57:22.869669 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.869624    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"]
Oct 30 13:57:22.870355 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870336    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main"
Oct 30 13:57:22.870401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870356    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main"
Oct 30 13:57:22.870401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870368    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main"
Oct 30 13:57:22.870401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870374    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main"
Oct 30 13:57:22.870401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870383    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b3533639-e576-418f-8c58-2e538f08ae0c" containerName="revoke-keys"
Oct 30 13:57:22.870401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870389    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3533639-e576-418f-8c58-2e538f08ae0c" containerName="revoke-keys"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870404    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5e0174df-681a-4867-834c-f790afda4824" containerName="revoke-keys"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870410    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0174df-681a-4867-834c-f790afda4824" containerName="revoke-keys"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870433    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870437    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870445    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870449    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870516    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efd7f65-f9e5-489c-b010-1ce9b04311eb" containerName="main"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870523    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3533639-e576-418f-8c58-2e538f08ae0c" containerName="revoke-keys"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870531    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0174df-681a-4867-834c-f790afda4824" containerName="revoke-keys"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870539    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0769aaca-8e70-4393-a400-5519741bc5e6" containerName="main"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870545    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b10838-9748-48ad-af2f-71502ed36435" containerName="main"
Oct 30 13:57:22.870557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.870553    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c48a685-e866-47d9-8673-673bb5bd0be3" containerName="main"
Oct 30 13:57:22.876294 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.876269    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:22.891167 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.891126    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:57:22.903162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.903126    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:57:22.903162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.903129    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:57:22.907550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.907513    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"]
Oct 30 13:57:22.949999 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.949945    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:22.950264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.950072    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:22.950264 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.950123    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pclr6\" (UniqueName: \"kubernetes.io/projected/2b710a37-6afa-4876-b9d7-eb6949f254fa-kube-api-access-pclr6\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:22.990932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:22.988481    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:57:23.050717 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:23.050669    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:23.050929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:23.050742    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:23.050929 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:23.050770    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclr6\" (UniqueName: \"kubernetes.io/projected/2b710a37-6afa-4876-b9d7-eb6949f254fa-kube-api-access-pclr6\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:23.051458 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:23.051431    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:23.053407 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:23.053380    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:23.074913 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:23.074872    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pclr6\" (UniqueName: \"kubernetes.io/projected/2b710a37-6afa-4876-b9d7-eb6949f254fa-kube-api-access-pclr6\") pod \"maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:23.194050 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:23.193960    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:23.505671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:23.505637    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"]
Oct 30 13:57:23.508501 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:57:23.508468    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b710a37_6afa_4876_b9d7_eb6949f254fa.slice/crio-b58c4e8e8e022dbdfb41abff6efc5dca0841c92256aee61e60901ef51a4d17a8 WatchSource:0}: Error finding container b58c4e8e8e022dbdfb41abff6efc5dca0841c92256aee61e60901ef51a4d17a8: Status 404 returned error can't find the container with id b58c4e8e8e022dbdfb41abff6efc5dca0841c92256aee61e60901ef51a4d17a8
Oct 30 13:57:24.034312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:24.034184    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz" event={"ID":"2b710a37-6afa-4876-b9d7-eb6949f254fa","Type":"ContainerStarted","Data":"5c81e112d447dbb77c146e9a8857bf52216f92c7fda2e37ed42a9c5ea206e001"}
Oct 30 13:57:24.034312 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:24.034273    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz" event={"ID":"2b710a37-6afa-4876-b9d7-eb6949f254fa","Type":"ContainerStarted","Data":"b58c4e8e8e022dbdfb41abff6efc5dca0841c92256aee61e60901ef51a4d17a8"}
Oct 30 13:57:24.077740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:24.077672    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz" podStartSLOduration=2.077653888 podStartE2EDuration="2.077653888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:24.076618566 +0000 UTC m=+5108.552338176" watchObservedRunningTime="2026-10-30 13:57:24.077653888 +0000 UTC m=+5108.553373497"
Oct 30 13:57:24.434023 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:24.433989    4993 scope.go:122] "RemoveContainer" containerID="95e20b4e57bb0be03468c0f2b261ddc3e17b3f61fa24d4e6aca95f0e3b49147b"
Oct 30 13:57:25.045021 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:25.044986    4993 generic.go:356] "Generic (PLEG): container finished" podID="2b710a37-6afa-4876-b9d7-eb6949f254fa" containerID="5c81e112d447dbb77c146e9a8857bf52216f92c7fda2e37ed42a9c5ea206e001" exitCode=0
Oct 30 13:57:25.045658 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:25.045139    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz" event={"ID":"2b710a37-6afa-4876-b9d7-eb6949f254fa","Type":"ContainerDied","Data":"5c81e112d447dbb77c146e9a8857bf52216f92c7fda2e37ed42a9c5ea206e001"}
Oct 30 13:57:26.220987 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.220951    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:26.285725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.285673    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-openshift-ca\") pod \"2b710a37-6afa-4876-b9d7-eb6949f254fa\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") "
Oct 30 13:57:26.285947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.285875    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b710a37-6afa-4876-b9d7-eb6949f254fa-kube-api-access-pclr6\" (UniqueName: \"kubernetes.io/projected/2b710a37-6afa-4876-b9d7-eb6949f254fa-kube-api-access-pclr6\") pod \"2b710a37-6afa-4876-b9d7-eb6949f254fa\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") "
Oct 30 13:57:26.286024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.285965    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-xks-ca\") pod \"2b710a37-6afa-4876-b9d7-eb6949f254fa\" (UID: \"2b710a37-6afa-4876-b9d7-eb6949f254fa\") "
Oct 30 13:57:26.286162 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.286132    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-openshift-ca" pod "2b710a37-6afa-4876-b9d7-eb6949f254fa" (UID: "2b710a37-6afa-4876-b9d7-eb6949f254fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:26.286422 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.286396    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:26.288269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.288235    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-xks-ca" pod "2b710a37-6afa-4876-b9d7-eb6949f254fa" (UID: "2b710a37-6afa-4876-b9d7-eb6949f254fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:26.288414 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.288368    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b710a37-6afa-4876-b9d7-eb6949f254fa-kube-api-access-pclr6" pod "2b710a37-6afa-4876-b9d7-eb6949f254fa" (UID: "2b710a37-6afa-4876-b9d7-eb6949f254fa"). InnerVolumeSpecName "kube-api-access-pclr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:26.387821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.387718    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pclr6\" (UniqueName: \"kubernetes.io/projected/2b710a37-6afa-4876-b9d7-eb6949f254fa-kube-api-access-pclr6\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:26.387821 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:26.387760    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b710a37-6afa-4876-b9d7-eb6949f254fa-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.062797 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:27.062763    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"
Oct 30 13:57:27.063033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:27.062797    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz" event={"ID":"2b710a37-6afa-4876-b9d7-eb6949f254fa","Type":"ContainerDied","Data":"b58c4e8e8e022dbdfb41abff6efc5dca0841c92256aee61e60901ef51a4d17a8"}
Oct 30 13:57:27.063033 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:27.062852    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58c4e8e8e022dbdfb41abff6efc5dca0841c92256aee61e60901ef51a4d17a8"
Oct 30 13:57:27.692289 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:27.692256    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"]
Oct 30 13:57:27.705981 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:27.705942    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"]
Oct 30 13:57:31.040127 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:31.040079    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"]
Oct 30 13:57:31.051955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:31.051909    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7760d8-8krch"]
Oct 30 13:57:31.691661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:31.691622    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0" path="/var/lib/kubelet/pods/051ad4fd-bf88-4f71-93d5-acc9b9c5c5a0/volumes"
Oct 30 13:57:36.276340 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.276299    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"]
Oct 30 13:57:36.276796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.276758    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2b710a37-6afa-4876-b9d7-eb6949f254fa" containerName="revoke-keys"
Oct 30 13:57:36.276796 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.276769    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b710a37-6afa-4876-b9d7-eb6949f254fa" containerName="revoke-keys"
Oct 30 13:57:36.276876 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.276840    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b710a37-6afa-4876-b9d7-eb6949f254fa" containerName="revoke-keys"
Oct 30 13:57:36.279953 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.279932    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.298947 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.298900    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:57:36.303291 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.303256    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"]
Oct 30 13:57:36.310974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.310931    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:57:36.310974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.310940    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:57:36.310974 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.310946    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:57:36.391260 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.391193    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.391440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.391377    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.391440 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.391416    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6nbx\" (UniqueName: \"kubernetes.io/projected/b93e01ac-d3fd-49ab-a204-d8582d301c72-kube-api-access-h6nbx\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.492746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.492700    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.492746 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.492745    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nbx\" (UniqueName: \"kubernetes.io/projected/b93e01ac-d3fd-49ab-a204-d8582d301c72-kube-api-access-h6nbx\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.493044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.492807    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.493467 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.493440    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.495331 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.495307    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.515901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.515864    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nbx\" (UniqueName: \"kubernetes.io/projected/b93e01ac-d3fd-49ab-a204-d8582d301c72-kube-api-access-h6nbx\") pod \"maas-api-revoke-sub-daa5e089-m4vrl\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.593901 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.593806    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:36.897150 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:36.897108    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"]
Oct 30 13:57:36.899122 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:57:36.899095    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93e01ac_d3fd_49ab_a204_d8582d301c72.slice/crio-2e71996f0d49c8d8fefbce0ee11655f9c01071a8d8a41e0af63bad665d1f34b3 WatchSource:0}: Error finding container 2e71996f0d49c8d8fefbce0ee11655f9c01071a8d8a41e0af63bad665d1f34b3: Status 404 returned error can't find the container with id 2e71996f0d49c8d8fefbce0ee11655f9c01071a8d8a41e0af63bad665d1f34b3
Oct 30 13:57:37.145083 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:37.145038    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl" event={"ID":"b93e01ac-d3fd-49ab-a204-d8582d301c72","Type":"ContainerStarted","Data":"2e71996f0d49c8d8fefbce0ee11655f9c01071a8d8a41e0af63bad665d1f34b3"}
Oct 30 13:57:38.154764 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:38.154671    4993 generic.go:356] "Generic (PLEG): container finished" podID="b93e01ac-d3fd-49ab-a204-d8582d301c72" containerID="ca77cdb3013a2306866186dbfefd592a410984d80e96414465c35fc2661e4e66" exitCode=0
Oct 30 13:57:38.155183 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:38.154773    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl" event={"ID":"b93e01ac-d3fd-49ab-a204-d8582d301c72","Type":"ContainerDied","Data":"ca77cdb3013a2306866186dbfefd592a410984d80e96414465c35fc2661e4e66"}
Oct 30 13:57:38.572463 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:57:38.572420    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:57:39.346781 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.346750    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:39.428864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.428759    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-xks-ca\") pod \"b93e01ac-d3fd-49ab-a204-d8582d301c72\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") "
Oct 30 13:57:39.428864 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.428822    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-openshift-ca\") pod \"b93e01ac-d3fd-49ab-a204-d8582d301c72\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") "
Oct 30 13:57:39.429133 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.429001    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93e01ac-d3fd-49ab-a204-d8582d301c72-kube-api-access-h6nbx\" (UniqueName: \"kubernetes.io/projected/b93e01ac-d3fd-49ab-a204-d8582d301c72-kube-api-access-h6nbx\") pod \"b93e01ac-d3fd-49ab-a204-d8582d301c72\" (UID: \"b93e01ac-d3fd-49ab-a204-d8582d301c72\") "
Oct 30 13:57:39.429242 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.429195    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-openshift-ca" pod "b93e01ac-d3fd-49ab-a204-d8582d301c72" (UID: "b93e01ac-d3fd-49ab-a204-d8582d301c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:39.429937 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.429906    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-openshift-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:39.431323 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.431294    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-xks-ca" pod "b93e01ac-d3fd-49ab-a204-d8582d301c72" (UID: "b93e01ac-d3fd-49ab-a204-d8582d301c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:39.432615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.432585    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93e01ac-d3fd-49ab-a204-d8582d301c72-kube-api-access-h6nbx" pod "b93e01ac-d3fd-49ab-a204-d8582d301c72" (UID: "b93e01ac-d3fd-49ab-a204-d8582d301c72"). InnerVolumeSpecName "kube-api-access-h6nbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:39.531487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.531443    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6nbx\" (UniqueName: \"kubernetes.io/projected/b93e01ac-d3fd-49ab-a204-d8582d301c72-kube-api-access-h6nbx\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:39.531487 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:39.531480    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93e01ac-d3fd-49ab-a204-d8582d301c72-maas-api-xks-ca\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:40.180432 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:40.180396    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl"
Oct 30 13:57:40.180638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:40.180397    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5e089-m4vrl" event={"ID":"b93e01ac-d3fd-49ab-a204-d8582d301c72","Type":"ContainerDied","Data":"2e71996f0d49c8d8fefbce0ee11655f9c01071a8d8a41e0af63bad665d1f34b3"}
Oct 30 13:57:40.180638 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:40.180543    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e71996f0d49c8d8fefbce0ee11655f9c01071a8d8a41e0af63bad665d1f34b3"
Oct 30 13:57:42.031063 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:42.031024    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"]
Oct 30 13:57:42.042649 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:42.042614    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd48803d-7cxxs"]
Oct 30 13:57:42.714749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:42.714708    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" containerID="cri-o://340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412" gracePeriod=45
Oct 30 13:57:42.727525 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:42.727485    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 13:57:42.730165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:42.730115    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main" containerID="cri-o://027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1" gracePeriod=45
Oct 30 13:57:43.017091 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.017061    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 13:57:43.049116 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.049080    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 13:57:43.170554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.170517    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d93d6785-85e4-493b-b87f-c73983fde374-tls-certs\" (UniqueName: \"kubernetes.io/secret/d93d6785-85e4-493b-b87f-c73983fde374-tls-certs\") pod \"d93d6785-85e4-493b-b87f-c73983fde374\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") "
Oct 30 13:57:43.170554 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.170559    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-tmp-dir\") pod \"28b70714-bfd7-4054-8f7f-21849ff3d386\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") "
Oct 30 13:57:43.170898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.170593    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-home\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-home\") pod \"d93d6785-85e4-493b-b87f-c73983fde374\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") "
Oct 30 13:57:43.170898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.170620    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-dshm\") pod \"28b70714-bfd7-4054-8f7f-21849ff3d386\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") "
Oct 30 13:57:43.170898 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.170880    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-tmp-dir" pod "28b70714-bfd7-4054-8f7f-21849ff3d386" (UID: "28b70714-bfd7-4054-8f7f-21849ff3d386"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:43.171073 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171051    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-tmp-dir\") pod \"d93d6785-85e4-493b-b87f-c73983fde374\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") "
Oct 30 13:57:43.171165 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171149    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-dshm\") pod \"d93d6785-85e4-493b-b87f-c73983fde374\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") "
Oct 30 13:57:43.171249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171185    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b70714-bfd7-4054-8f7f-21849ff3d386-tls-certs\" (UniqueName: \"kubernetes.io/secret/28b70714-bfd7-4054-8f7f-21849ff3d386-tls-certs\") pod \"28b70714-bfd7-4054-8f7f-21849ff3d386\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") "
Oct 30 13:57:43.171249 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171195    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-tmp-dir" pod "d93d6785-85e4-493b-b87f-c73983fde374" (UID: "d93d6785-85e4-493b-b87f-c73983fde374"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:43.171330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171251    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-home" pod "d93d6785-85e4-493b-b87f-c73983fde374" (UID: "d93d6785-85e4-493b-b87f-c73983fde374"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:43.171330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171264    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-model-cache\") pod \"d93d6785-85e4-493b-b87f-c73983fde374\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") "
Oct 30 13:57:43.171405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171366    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93d6785-85e4-493b-b87f-c73983fde374-kube-api-access-p5lzv\" (UniqueName: \"kubernetes.io/projected/d93d6785-85e4-493b-b87f-c73983fde374-kube-api-access-p5lzv\") pod \"d93d6785-85e4-493b-b87f-c73983fde374\" (UID: \"d93d6785-85e4-493b-b87f-c73983fde374\") "
Oct 30 13:57:43.171405 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171395    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-model-cache\") pod \"28b70714-bfd7-4054-8f7f-21849ff3d386\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") "
Oct 30 13:57:43.171513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171422    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b70714-bfd7-4054-8f7f-21849ff3d386-kube-api-access-hbh7w\" (UniqueName: \"kubernetes.io/projected/28b70714-bfd7-4054-8f7f-21849ff3d386-kube-api-access-hbh7w\") pod \"28b70714-bfd7-4054-8f7f-21849ff3d386\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") "
Oct 30 13:57:43.171513 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.171473    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-home\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-home\") pod \"28b70714-bfd7-4054-8f7f-21849ff3d386\" (UID: \"28b70714-bfd7-4054-8f7f-21849ff3d386\") "
Oct 30 13:57:43.172654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.172336    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-model-cache" pod "d93d6785-85e4-493b-b87f-c73983fde374" (UID: "d93d6785-85e4-493b-b87f-c73983fde374"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:43.172654 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.172605    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-home" pod "28b70714-bfd7-4054-8f7f-21849ff3d386" (UID: "28b70714-bfd7-4054-8f7f-21849ff3d386"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:43.172890 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.172862    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.172972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.172898    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.172972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.172915    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.172972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.172929    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-tmp-dir\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.172972 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.172941    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-home\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.173725 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.173696    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-dshm" pod "d93d6785-85e4-493b-b87f-c73983fde374" (UID: "d93d6785-85e4-493b-b87f-c73983fde374"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:43.173895 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.173803    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-dshm" pod "28b70714-bfd7-4054-8f7f-21849ff3d386" (UID: "28b70714-bfd7-4054-8f7f-21849ff3d386"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:43.174362 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.174315    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b70714-bfd7-4054-8f7f-21849ff3d386-tls-certs" pod "28b70714-bfd7-4054-8f7f-21849ff3d386" (UID: "28b70714-bfd7-4054-8f7f-21849ff3d386"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:43.174854 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.174823    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-model-cache" pod "28b70714-bfd7-4054-8f7f-21849ff3d386" (UID: "28b70714-bfd7-4054-8f7f-21849ff3d386"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:43.175252 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.175193    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d93d6785-85e4-493b-b87f-c73983fde374-tls-certs" pod "d93d6785-85e4-493b-b87f-c73983fde374" (UID: "d93d6785-85e4-493b-b87f-c73983fde374"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:43.175547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.175519    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93d6785-85e4-493b-b87f-c73983fde374-kube-api-access-p5lzv" pod "d93d6785-85e4-493b-b87f-c73983fde374" (UID: "d93d6785-85e4-493b-b87f-c73983fde374"). InnerVolumeSpecName "kube-api-access-p5lzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:43.175615 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.175568    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b70714-bfd7-4054-8f7f-21849ff3d386-kube-api-access-hbh7w" pod "28b70714-bfd7-4054-8f7f-21849ff3d386" (UID: "28b70714-bfd7-4054-8f7f-21849ff3d386"). InnerVolumeSpecName "kube-api-access-hbh7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:43.207032 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.206993    4993 generic.go:356] "Generic (PLEG): container finished" podID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerID="027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1" exitCode=0
Oct 30 13:57:43.207257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.207047    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"
Oct 30 13:57:43.207257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.207074    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" event={"ID":"28b70714-bfd7-4054-8f7f-21849ff3d386","Type":"ContainerDied","Data":"027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1"}
Oct 30 13:57:43.207257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.207118    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz" event={"ID":"28b70714-bfd7-4054-8f7f-21849ff3d386","Type":"ContainerDied","Data":"e0c0fe14def56737820f0fecef585f05b6b9b14e027cbe9058d5c072d55cd276"}
Oct 30 13:57:43.207257 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.207136    4993 scope.go:122] "RemoveContainer" containerID="027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1"
Oct 30 13:57:43.208827 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.208799    4993 generic.go:356] "Generic (PLEG): container finished" podID="d93d6785-85e4-493b-b87f-c73983fde374" containerID="340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412" exitCode=0
Oct 30 13:57:43.208933 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.208841    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"
Oct 30 13:57:43.209000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.208931    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" event={"ID":"d93d6785-85e4-493b-b87f-c73983fde374","Type":"ContainerDied","Data":"340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412"}
Oct 30 13:57:43.209000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.208972    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9" event={"ID":"d93d6785-85e4-493b-b87f-c73983fde374","Type":"ContainerDied","Data":"4475fefa4ef4e10bf600d457cad69a0d35382d10c39dae0e8585ed031f723529"}
Oct 30 13:57:43.222397 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.222253    4993 scope.go:122] "RemoveContainer" containerID="027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1"
Oct 30 13:57:43.222731 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:57:43.222628    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1\": container with ID starting with 027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1 not found: ID does not exist" containerID="027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1"
Oct 30 13:57:43.222731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.222671    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1"} err="failed to get container status \"027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1\": rpc error: code = NotFound desc = could not find container \"027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1\": container with ID starting with 027f1ab3b8897ace8ba8270f737e2fb5d9469e9956e19fe1479a2f06d17200e1 not found: ID does not exist"
Oct 30 13:57:43.222731 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.222695    4993 scope.go:122] "RemoveContainer" containerID="340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412"
Oct 30 13:57:43.233411 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.233381    4993 scope.go:122] "RemoveContainer" containerID="340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412"
Oct 30 13:57:43.233720 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:57:43.233693    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412\": container with ID starting with 340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412 not found: ID does not exist" containerID="340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412"
Oct 30 13:57:43.233780 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.233734    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412"} err="failed to get container status \"340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412\": rpc error: code = NotFound desc = could not find container \"340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412\": container with ID starting with 340cecb760c98aeba6acfb36c456967426a328b5e8c9723765e5f3633c790412 not found: ID does not exist"
Oct 30 13:57:43.273147 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.273050    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"]
Oct 30 13:57:43.273670 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.273636    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5lzv\" (UniqueName: \"kubernetes.io/projected/d93d6785-85e4-493b-b87f-c73983fde374-kube-api-access-p5lzv\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.273825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.273673    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-model-cache\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.273825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.273688    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbh7w\" (UniqueName: \"kubernetes.io/projected/28b70714-bfd7-4054-8f7f-21849ff3d386-kube-api-access-hbh7w\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.273825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.273704    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d93d6785-85e4-493b-b87f-c73983fde374-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.273825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.273719    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b70714-bfd7-4054-8f7f-21849ff3d386-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.273825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.273731    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d93d6785-85e4-493b-b87f-c73983fde374-dshm\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.273825 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.273743    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b70714-bfd7-4054-8f7f-21849ff3d386-tls-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:57:43.277191 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.277161    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/simulator-w3-551dd1-kserve-5948746798-drxvz"]
Oct 30 13:57:43.310088 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.310048    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"]
Oct 30 13:57:43.314157 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.314121    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-w3-551dd1-kserve-565f4c4db-8snn9"]
Oct 30 13:57:43.691102 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.691022    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" path="/var/lib/kubelet/pods/28b70714-bfd7-4054-8f7f-21849ff3d386/volumes"
Oct 30 13:57:43.691522 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.691506    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae167aa9-d7d6-4a75-a211-c16a83b3e0e2" path="/var/lib/kubelet/pods/ae167aa9-d7d6-4a75-a211-c16a83b3e0e2/volumes"
Oct 30 13:57:43.692000 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:43.691985    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93d6785-85e4-493b-b87f-c73983fde374" path="/var/lib/kubelet/pods/d93d6785-85e4-493b-b87f-c73983fde374/volumes"
Oct 30 13:57:52.062740 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:52.062698    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"]
Oct 30 13:57:52.075636 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:52.075595    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-551dd1-bpbpz"]
Oct 30 13:57:53.693051 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:53.693019    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b710a37-6afa-4876-b9d7-eb6949f254fa" path="/var/lib/kubelet/pods/2b710a37-6afa-4876-b9d7-eb6949f254fa/volumes"
Oct 30 13:57:59.053722 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:59.052997    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"]
Oct 30 13:57:59.054671 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:59.054634    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy" containerID="cri-o://8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f" gracePeriod=30
Oct 30 13:57:59.740991 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:59.740949    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:57:59.741233 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:57:59.741014    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:58:00.147611 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:00.147514    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6956fb6d85-krc2x_862929ad-91d4-47d8-8740-21c32f987f11/authorino/0.log"
Oct 30 13:58:00.390588 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:00.390552    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7vzgf_8b08b699-9669-481c-9622-e82c35b0e729/manager/0.log"
Oct 30 13:58:00.783859 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:00.783828    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dn49h_cd04f389-4aae-4a44-882c-be7855d43b44/kuadrant-console-plugin/0.log"
Oct 30 13:58:01.167593 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:01.167504    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7hpfd_0aff0921-3696-4f79-8380-f4bd418fde32/manager/0.log"
Oct 30 13:58:01.601870 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:01.601831    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tvfdn_9508df2d-3fcd-41aa-8559-2def2056346c/manager/0.log"
Oct 30 13:58:02.618187 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:02.618144    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mp6wv_831c60d3-728e-4e13-8e6b-4b42d922c47e/manager/0.log"
Oct 30 13:58:03.098692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:03.098651    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ln8c_0ae3b9f9-8971-4208-8518-b25746b98186/manager/0.log"
Oct 30 13:58:03.942163 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:03.942131    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bt6xn_700e35bb-15f5-4740-8132-fa48da68e589/server/0.log"
Oct 30 13:58:04.150261 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.150200    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm28s_85eadb46-be52-4e6d-8813-46bb7b155a39/manager/0.log"
Oct 30 13:58:04.341027 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.340989    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 13:58:04.367783 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.367745    4993 generic.go:356] "Generic (PLEG): container finished" podID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerID="8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f" exitCode=0
Oct 30 13:58:04.368001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.367822    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"
Oct 30 13:58:04.368001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.367836    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" event={"ID":"cb7857c6-5221-4532-ae92-fa36943e76b1","Type":"ContainerDied","Data":"8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f"}
Oct 30 13:58:04.368001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.367893    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs" event={"ID":"cb7857c6-5221-4532-ae92-fa36943e76b1","Type":"ContainerDied","Data":"6d7dbb2ab7d7893f67cd3c21b9e3b9f4e9b0eda397cc0d3d57b1f068e7b6a9e2"}
Oct 30 13:58:04.368001 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.367919    4993 scope.go:122] "RemoveContainer" containerID="8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f"
Oct 30 13:58:04.380798 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.380743    4993 scope.go:122] "RemoveContainer" containerID="8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f"
Oct 30 13:58:04.381249 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:58:04.381221    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f\": container with ID starting with 8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f not found: ID does not exist" containerID="8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f"
Oct 30 13:58:04.381408 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.381263    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f"} err="failed to get container status \"8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f\": rpc error: code = NotFound desc = could not find container \"8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f\": container with ID starting with 8c9e215f54b545db5e3620316af92a7ccb8679dc392a8dff01e4f80c68f8850f not found: ID does not exist"
Oct 30 13:58:04.382776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.382746    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-data\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.383008 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.382985    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-podinfo\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.383152 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.383132    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-token\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-token\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.383350 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.383330    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-credential-socket\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.383531 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.383510    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-kube-api-access-47mg2\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-kube-api-access-47mg2\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.383663 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.383644    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-certs\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.383837 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.383819    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7857c6-5221-4532-ae92-fa36943e76b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7857c6-5221-4532-ae92-fa36943e76b1-istiod-ca-cert\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.384042 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.384024    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-envoy\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.384192 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.384176    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-socket\") pod \"cb7857c6-5221-4532-ae92-fa36943e76b1\" (UID: \"cb7857c6-5221-4532-ae92-fa36943e76b1\") "
Oct 30 13:58:04.387749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.384618    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-data" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:04.387749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.385197    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-socket" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:04.387749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.385403    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-certs" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:04.387749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.385673    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7857c6-5221-4532-ae92-fa36943e76b1-istiod-ca-cert" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:04.387749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.385598    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.387749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.385783    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-data\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.387749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.385834    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-workload-certs\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.387749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.387049    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-credential-socket" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:04.410200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.408916    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-kube-api-access-47mg2" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "kube-api-access-47mg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:04.410787 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.410752    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-token" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:04.412775 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.412743    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-envoy" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:04.413200 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.413176    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-podinfo" pod "cb7857c6-5221-4532-ae92-fa36943e76b1" (UID: "cb7857c6-5221-4532-ae92-fa36943e76b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:58:04.486977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.486867    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-podinfo\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.486977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.486918    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-token\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.486977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.486928    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-credential-socket\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.486977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.486938    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mg2\" (UniqueName: \"kubernetes.io/projected/cb7857c6-5221-4532-ae92-fa36943e76b1-kube-api-access-47mg2\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.486977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.486947    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7857c6-5221-4532-ae92-fa36943e76b1-istiod-ca-cert\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.486977 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.486957    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7857c6-5221-4532-ae92-fa36943e76b1-istio-envoy\") on node \"ip-10-0-135-24.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.714320 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.714275    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"]
Oct 30 13:58:04.725794 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:04.725751    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-lb7xs"]
Oct 30 13:58:05.158358 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:05.158324    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6956fb6d85-krc2x_862929ad-91d4-47d8-8740-21c32f987f11/authorino/0.log"
Oct 30 13:58:05.692044 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:05.692010    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" path="/var/lib/kubelet/pods/cb7857c6-5221-4532-ae92-fa36943e76b1/volumes"
Oct 30 13:58:08.569350 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:58:08.569303    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:58:16.010034 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:16.010003    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mp6wv_831c60d3-728e-4e13-8e6b-4b42d922c47e/manager/0.log"
Oct 30 13:58:16.446453 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:16.446333    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ln8c_0ae3b9f9-8971-4208-8518-b25746b98186/manager/0.log"
Oct 30 13:58:17.032879 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:17.032839    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"]
Oct 30 13:58:17.044805 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:17.044765    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c8aa88-5tc8l"]
Oct 30 13:58:17.221958 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:17.221896    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bt6xn_700e35bb-15f5-4740-8132-fa48da68e589/server/0.log"
Oct 30 13:58:17.425357 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:17.425275    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm28s_85eadb46-be52-4e6d-8813-46bb7b155a39/manager/0.log"
Oct 30 13:58:17.689829 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:17.689748    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aaf2d54-a366-432d-ad2e-0475a16412cb" path="/var/lib/kubelet/pods/6aaf2d54-a366-432d-ad2e-0475a16412cb/volumes"
Oct 30 13:58:18.659750 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:18.659713    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0bd9da67-cshsq_06f6482f-5cc0-44d7-93d0-7e01944205b7/revoke-keys/0.log"
Oct 30 13:58:18.857428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:18.857389    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0c9ad015-6l9pq_748ca8b1-9602-4a02-a162-d222afb56e5e/revoke-keys/0.log"
Oct 30 13:58:19.055790 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:19.055753    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0cbbd780-c78t5_a32d9c54-c294-4d00-ac1c-31829f7a67ef/revoke-keys/0.log"
Oct 30 13:58:19.242993 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:19.242957    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-15d99453-2j745_440c9ea0-5b0f-479f-9636-fe573c3f2dfc/revoke-keys/0.log"
Oct 30 13:58:19.633346 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:19.633318    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-34271737-w8rtf_a4aa7a8b-d07b-4035-88d1-16a9dd702412/revoke-keys/0.log"
Oct 30 13:58:19.820330 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:19.820299    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-462560f3-ts2q5_0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807/revoke-keys/0.log"
Oct 30 13:58:19.999622 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:19.999588    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4ec788d2-tcv7x_80acf968-3c72-4f2a-bb7d-afb7bef1d820/revoke-keys/0.log"
Oct 30 13:58:20.381702 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:20.381620    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-835a9ed8-b6bm9_27c93a6d-c980-4497-8ce5-826281996589/revoke-keys/0.log"
Oct 30 13:58:20.786856 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:20.786823    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bd0da700-sqlg9_078c5fa3-22e6-4ef2-acc1-07fbc3b32fba/revoke-keys/0.log"
Oct 30 13:58:20.975808 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:20.975779    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bf8ccd0e-lh9vh_eca05d59-a006-4ff1-a844-3edd9761878b/revoke-keys/0.log"
Oct 30 13:58:21.161946 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:21.161854    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-daa5e089-m4vrl_b93e01ac-d3fd-49ab-a204-d8582d301c72/revoke-keys/0.log"
Oct 30 13:58:21.372065 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:21.372030    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-df82330b-xlqmn_20c51228-21bd-4104-8fa8-eafb0954f18b/revoke-keys/0.log"
Oct 30 13:58:21.561457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:21.561420    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e16b61b5-7pzkh_b3533639-e576-418f-8c58-2e538f08ae0c/revoke-keys/0.log"
Oct 30 13:58:21.751778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:21.751747    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ef524820-6z89w_4d456c31-7321-471e-bf1d-90abe178e46e/revoke-keys/0.log"
Oct 30 13:58:23.028802 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:23.028762    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"]
Oct 30 13:58:23.039886 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:23.039850    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15d99453-2j745"]
Oct 30 13:58:23.519558 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:23.519521    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6956fb6d85-krc2x_862929ad-91d4-47d8-8740-21c32f987f11/authorino/0.log"
Oct 30 13:58:23.691138 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:23.691102    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440c9ea0-5b0f-479f-9636-fe573c3f2dfc" path="/var/lib/kubelet/pods/440c9ea0-5b0f-479f-9636-fe573c3f2dfc/volumes"
Oct 30 13:58:23.759692 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:23.759656    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7vzgf_8b08b699-9669-481c-9622-e82c35b0e729/manager/0.log"
Oct 30 13:58:24.030365 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:24.030327    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"]
Oct 30 13:58:24.042159 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:24.042119    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef524820-6z89w"]
Oct 30 13:58:24.173547 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:24.173513    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dn49h_cd04f389-4aae-4a44-882c-be7855d43b44/kuadrant-console-plugin/0.log"
Oct 30 13:58:24.498849 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:24.498809    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7hpfd_0aff0921-3696-4f79-8380-f4bd418fde32/manager/0.log"
Oct 30 13:58:24.628846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:24.628811    4993 scope.go:122] "RemoveContainer" containerID="f1c271486b76da2b64cde1198541a5c6d8bd27ce38a2544633a1e86a360af25f"
Oct 30 13:58:24.640428 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:24.640391    4993 scope.go:122] "RemoveContainer" containerID="b018a380f05e726f76fedd6252c37283d62bd45bcd5c00518a2bff831e2e0f06"
Oct 30 13:58:24.650894 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:24.650867    4993 scope.go:122] "RemoveContainer" containerID="97b00bab8779753009000e09b9dd58b26217bbaaf11b7fe8f5e5f87e6b571821"
Oct 30 13:58:24.938776 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:24.938682    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tvfdn_9508df2d-3fcd-41aa-8559-2def2056346c/manager/0.log"
Oct 30 13:58:25.690902 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:25.690854    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d456c31-7321-471e-bf1d-90abe178e46e" path="/var/lib/kubelet/pods/4d456c31-7321-471e-bf1d-90abe178e46e/volumes"
Oct 30 13:58:25.727290 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:25.727251    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5mf69c_38996bb0-1dd5-449a-b542-814106c08ff6/istio-proxy/0.log"
Oct 30 13:58:25.931371 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:25.931334    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mnrt8_c2c5cff9-185e-482f-8a61-b2ec954a3c14/discovery/0.log"
Oct 30 13:58:26.558269 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:26.558183    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-czngj_d662a085-9c77-4c7c-82e1-de6c7101715d/istio-proxy/0.log"
Oct 30 13:58:27.362846 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:27.362811    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgkz8_d33694bd-a804-4182-8d4b-e30156066d8f/istio-proxy/1.log"
Oct 30 13:58:27.584216 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:27.584171    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh_81bb41a5-08e3-4e05-baee-c015d7239d96/istio-proxy/0.log"
Oct 30 13:58:28.028684 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:28.028648    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"]
Oct 30 13:58:28.040482 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:28.040444    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462560f3-ts2q5"]
Oct 30 13:58:28.170354 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:28.170314    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77bb4759d7-ztvnj_51f4f667-c671-472c-ae68-beb10d17b645/router/0.log"
Oct 30 13:58:28.608793 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:28.608758    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgkz8_d33694bd-a804-4182-8d4b-e30156066d8f/istio-proxy/0.log"
Oct 30 13:58:29.690173 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:29.690138    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807" path="/var/lib/kubelet/pods/0b662f71-a8fc-4b3c-bcdd-fbb7f4ea9807/volumes"
Oct 30 13:58:38.568858 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:58:38.568813    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:58:39.033333 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:39.033291    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"]
Oct 30 13:58:39.044707 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:39.044671    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ad015-6l9pq"]
Oct 30 13:58:39.690334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:39.690298    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748ca8b1-9602-4a02-a162-d222afb56e5e" path="/var/lib/kubelet/pods/748ca8b1-9602-4a02-a162-d222afb56e5e/volumes"
Oct 30 13:58:46.815883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:46.815841    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s4gm9_f25010f1-45b5-4994-91fa-7de15f04fb08/global-pull-secret-syncer/0.log"
Oct 30 13:58:47.028416 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:47.028386    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-829vl_0b3f7eca-7b44-46c3-b1fd-d9bb4f848539/connection-checker/0.log"
Oct 30 13:58:47.194235 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:47.194132    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pbb9n_c5bb30ba-a903-4c97-80a8-6a1f398b6664/connection-checker/0.log"
Oct 30 13:58:47.361377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:47.361339    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vwdb5_090d2edb-0fb8-4fa0-8d91-d866c06226b3/connection-checker/0.log"
Oct 30 13:58:47.851867 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:47.851828    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tnrf7_3dd41889-9433-4fdb-8e05-52ea68e32536/konnectivity-agent/0.log"
Oct 30 13:58:48.376919 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:48.376878    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-24.ec2.internal_fdd80fd13fc4b06ccce2af5848f9d942/haproxy/0.log"
Oct 30 13:58:50.043234 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:50.043183    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"]
Oct 30 13:58:50.054683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:50.054639    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cbbd780-c78t5"]
Oct 30 13:58:51.690248 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:51.690186    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32d9c54-c294-4d00-ac1c-31829f7a67ef" path="/var/lib/kubelet/pods/a32d9c54-c294-4d00-ac1c-31829f7a67ef/volumes"
Oct 30 13:58:54.655334 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:54.655291    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6956fb6d85-krc2x_862929ad-91d4-47d8-8740-21c32f987f11/authorino/0.log"
Oct 30 13:58:54.757891 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:54.757851    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-7vzgf_8b08b699-9669-481c-9622-e82c35b0e729/manager/0.log"
Oct 30 13:58:55.026605 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:55.026569    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dn49h_cd04f389-4aae-4a44-882c-be7855d43b44/kuadrant-console-plugin/0.log"
Oct 30 13:58:55.402335 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:55.402245    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7hpfd_0aff0921-3696-4f79-8380-f4bd418fde32/manager/0.log"
Oct 30 13:58:56.117443 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:56.117359    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tvfdn_9508df2d-3fcd-41aa-8559-2def2056346c/manager/0.log"
Oct 30 13:58:57.903578 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.903535    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"]
Oct 30 13:58:57.904024 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904009    4993 cpu_manager.go:414] "containerMap: removing container" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main"
Oct 30 13:58:57.904078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904025    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main"
Oct 30 13:58:57.904078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904034    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main"
Oct 30 13:58:57.904078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904039    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main"
Oct 30 13:58:57.904078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904057    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b93e01ac-d3fd-49ab-a204-d8582d301c72" containerName="revoke-keys"
Oct 30 13:58:57.904078 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904066    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93e01ac-d3fd-49ab-a204-d8582d301c72" containerName="revoke-keys"
Oct 30 13:58:57.904285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904088    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy"
Oct 30 13:58:57.904285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904121    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy"
Oct 30 13:58:57.904285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904247    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7857c6-5221-4532-ae92-fa36943e76b1" containerName="istio-proxy"
Oct 30 13:58:57.904285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904267    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b70714-bfd7-4054-8f7f-21849ff3d386" containerName="main"
Oct 30 13:58:57.904285 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904277    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93d6785-85e4-493b-b87f-c73983fde374" containerName="main"
Oct 30 13:58:57.904504 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.904288    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93e01ac-d3fd-49ab-a204-d8582d301c72" containerName="revoke-keys"
Oct 30 13:58:57.908695 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.908677    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:57.926068 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.926034    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z45hd\"/\"kube-root-ca.crt\""
Oct 30 13:58:57.929381 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.929350    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"]
Oct 30 13:58:57.938098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.938066    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z45hd\"/\"openshift-service-ca.crt\""
Oct 30 13:58:57.938098 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.938076    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z45hd\"/\"default-dockercfg-ln669\""
Oct 30 13:58:57.968353 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.968310    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-sys\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:57.968564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.968379    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-lib-modules\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:57.968564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.968466    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-podres\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:57.968564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.968500    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-proc\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:57.968564 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:57.968528    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrnf\" (UniqueName: \"kubernetes.io/projected/d5893af0-f6a7-454d-95c9-8e290f45bf9e-kube-api-access-zrrnf\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069379 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069337    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-lib-modules\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069392    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-podres\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069418    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-proc\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069441    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrnf\" (UniqueName: \"kubernetes.io/projected/d5893af0-f6a7-454d-95c9-8e290f45bf9e-kube-api-access-zrrnf\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069482    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-sys\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069497    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-lib-modules\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069538    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-podres\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069562    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-sys\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.069590 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.069549    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5893af0-f6a7-454d-95c9-8e290f45bf9e-proc\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.092253 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.092221    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrnf\" (UniqueName: \"kubernetes.io/projected/d5893af0-f6a7-454d-95c9-8e290f45bf9e-kube-api-access-zrrnf\") pod \"perf-node-gather-daemonset-djg4j\" (UID: \"d5893af0-f6a7-454d-95c9-8e290f45bf9e\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.226990 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.226956    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.528841 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.528805    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"]
Oct 30 13:58:58.530929 ip-10-0-135-24 kubenswrapper[4993]: W1030 13:58:58.530888    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd5893af0_f6a7_454d_95c9_8e290f45bf9e.slice/crio-240f28d76ad582ac7de9a207507b27de7ce8160ab8a31c7b9260ac092e95e791 WatchSource:0}: Error finding container 240f28d76ad582ac7de9a207507b27de7ce8160ab8a31c7b9260ac092e95e791: Status 404 returned error can't find the container with id 240f28d76ad582ac7de9a207507b27de7ce8160ab8a31c7b9260ac092e95e791
Oct 30 13:58:58.707855 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.707817    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-8pxzl_fd1c2c87-7cd8-4a13-a4a0-ff95ddffdd20/cluster-monitoring-operator/0.log"
Oct 30 13:58:58.757678 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.757631    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j" event={"ID":"d5893af0-f6a7-454d-95c9-8e290f45bf9e","Type":"ContainerStarted","Data":"421db2161e34b86946b970382fddbcdf92edc6cfbade5f71e1692f1ad4f14938"}
Oct 30 13:58:58.757899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.757699    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j" event={"ID":"d5893af0-f6a7-454d-95c9-8e290f45bf9e","Type":"ContainerStarted","Data":"240f28d76ad582ac7de9a207507b27de7ce8160ab8a31c7b9260ac092e95e791"}
Oct 30 13:58:58.757899 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.757825    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:58:58.801833 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:58.801723    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j" podStartSLOduration=1.80170369 podStartE2EDuration="1.80170369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:58.798637949 +0000 UTC m=+5203.274357556" watchObservedRunningTime="2026-10-30 13:58:58.80170369 +0000 UTC m=+5203.277423341"
Oct 30 13:58:59.443624 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:59.443578    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mp6wv_831c60d3-728e-4e13-8e6b-4b42d922c47e/manager/0.log"
Oct 30 13:58:59.475557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:59.475519    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mp6wv_831c60d3-728e-4e13-8e6b-4b42d922c47e/manager/0.log"
Oct 30 13:58:59.544401 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:59.544359    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pqg2_e97fe730-6f69-4c33-b12f-280022e5303b/node-exporter/0.log"
Oct 30 13:58:59.666566 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:59.666529    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pqg2_e97fe730-6f69-4c33-b12f-280022e5303b/kube-rbac-proxy/0.log"
Oct 30 13:58:59.800492 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:58:59.800457    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pqg2_e97fe730-6f69-4c33-b12f-280022e5303b/init-textfile/0.log"
Oct 30 13:59:00.500304 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:00.500256    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ln8c_0ae3b9f9-8971-4208-8518-b25746b98186/manager/0.log"
Oct 30 13:59:00.525315 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:00.525272    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ln8c_0ae3b9f9-8971-4208-8518-b25746b98186/manager/0.log"
Oct 30 13:59:01.154782 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:01.154745    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bt6xn_700e35bb-15f5-4740-8132-fa48da68e589/server/0.log"
Oct 30 13:59:01.172749 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:01.172722    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bt6xn_700e35bb-15f5-4740-8132-fa48da68e589/server/0.log"
Oct 30 13:59:03.015014 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:03.014970    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm28s_85eadb46-be52-4e6d-8813-46bb7b155a39/manager/0.log"
Oct 30 13:59:03.022708 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:03.022670    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm28s_85eadb46-be52-4e6d-8813-46bb7b155a39/manager/0.log"
Oct 30 13:59:04.764568 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:04.764532    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mp6wv_831c60d3-728e-4e13-8e6b-4b42d922c47e/manager/0.log"
Oct 30 13:59:04.777071 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:04.777045    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-djg4j"
Oct 30 13:59:04.912761 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:04.912722    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mp6wv_831c60d3-728e-4e13-8e6b-4b42d922c47e/manager/0.log"
Oct 30 13:59:05.584561 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:05.584521    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-87dsg_5d6d13bc-3169-4d2c-830c-66837db3faf4/networking-console-plugin/0.log"
Oct 30 13:59:05.787374 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:05.787328    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ln8c_0ae3b9f9-8971-4208-8518-b25746b98186/manager/0.log"
Oct 30 13:59:05.962883 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:05.962795    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ln8c_0ae3b9f9-8971-4208-8518-b25746b98186/manager/0.log"
Oct 30 13:59:06.312026 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:06.311996    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bt6xn_700e35bb-15f5-4740-8132-fa48da68e589/server/0.log"
Oct 30 13:59:06.965321 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:06.965290    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bt6xn_700e35bb-15f5-4740-8132-fa48da68e589/server/0.log"
Oct 30 13:59:07.072922 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:07.072884    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/3.log"
Oct 30 13:59:07.080457 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:07.080423    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-lq5sc_c0e41a51-8711-47dd-a04d-af098a4d7fcb/console-operator/4.log"
Oct 30 13:59:07.951778 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:07.951742    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm28s_85eadb46-be52-4e6d-8813-46bb7b155a39/manager/0.log"
Oct 30 13:59:08.565668 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:08.565626    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm28s_85eadb46-be52-4e6d-8813-46bb7b155a39/manager/0.log"
Oct 30 13:59:08.575256 ip-10-0-135-24 kubenswrapper[4993]: E1030 13:59:08.575177    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gl9mb" containerName="extractor"
Oct 30 13:59:09.744479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:09.744443    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-4t2tb_b02a9da2-ad91-4bac-8177-c851dc4a37d3/volume-data-source-validator/0.log"
Oct 30 13:59:09.995824 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:09.995740    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mp6wv_831c60d3-728e-4e13-8e6b-4b42d922c47e/manager/0.log"
Oct 30 13:59:10.559277 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:10.559239    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mp6wv_831c60d3-728e-4e13-8e6b-4b42d922c47e/manager/0.log"
Oct 30 13:59:10.847194 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:10.847069    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ln8c_0ae3b9f9-8971-4208-8518-b25746b98186/manager/0.log"
Oct 30 13:59:11.395557 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:11.395518    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2ln8c_0ae3b9f9-8971-4208-8518-b25746b98186/manager/0.log"
Oct 30 13:59:11.439313 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:11.439275    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bt6xn_700e35bb-15f5-4740-8132-fa48da68e589/server/0.log"
Oct 30 13:59:11.963171 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:11.963130    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bt6xn_700e35bb-15f5-4740-8132-fa48da68e589/server/0.log"
Oct 30 13:59:12.863635 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:12.863586    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qngfl_179c5618-551b-4e6e-b0d3-5fa27708fb1f/dns/0.log"
Oct 30 13:59:12.983104 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:12.983067    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qngfl_179c5618-551b-4e6e-b0d3-5fa27708fb1f/kube-rbac-proxy/0.log"
Oct 30 13:59:13.001641 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:13.001603    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5mf69c_38996bb0-1dd5-449a-b542-814106c08ff6/istio-proxy/0.log"
Oct 30 13:59:13.079316 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:13.079280    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm28s_85eadb46-be52-4e6d-8813-46bb7b155a39/manager/0.log"
Oct 30 13:59:13.381344 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:13.381306    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-98w6d_ee1dad4b-c7ea-4f2a-9dfc-c5504675e4e4/dns-node-resolver/0.log"
Oct 30 13:59:13.470932 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:13.470898    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mnrt8_c2c5cff9-185e-482f-8a61-b2ec954a3c14/discovery/0.log"
Oct 30 13:59:13.686196 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:13.686092    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-czngj_d662a085-9c77-4c7c-82e1-de6c7101715d/istio-proxy/0.log"
Oct 30 13:59:13.697955 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:13.697925    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm28s_85eadb46-be52-4e6d-8813-46bb7b155a39/manager/0.log"
Oct 30 13:59:14.037141 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:14.037107    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgkz8_d33694bd-a804-4182-8d4b-e30156066d8f/istio-proxy/0.log"
Oct 30 13:59:14.038377 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:14.038356    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgkz8_d33694bd-a804-4182-8d4b-e30156066d8f/istio-proxy/1.log"
Oct 30 13:59:14.075607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:14.075570    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh_81bb41a5-08e3-4e05-baee-c015d7239d96/istio-proxy/0.log"
Oct 30 13:59:14.285842 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:14.285806    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77bb4759d7-ztvnj_51f4f667-c671-472c-ae68-beb10d17b645/router/0.log"
Oct 30 13:59:15.136186 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:15.136155    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wjpmw_46fbee35-1719-4228-a543-874c0d3f4b4b/node-ca/0.log"
Oct 30 13:59:15.282865 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:15.282834    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5mf69c_38996bb0-1dd5-449a-b542-814106c08ff6/istio-proxy/0.log"
Oct 30 13:59:15.587062 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:15.587022    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mnrt8_c2c5cff9-185e-482f-8a61-b2ec954a3c14/discovery/0.log"
Oct 30 13:59:15.758683 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:15.758649    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5mf69c_38996bb0-1dd5-449a-b542-814106c08ff6/istio-proxy/0.log"
Oct 30 13:59:15.764916 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:15.764883    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-czngj_d662a085-9c77-4c7c-82e1-de6c7101715d/istio-proxy/0.log"
Oct 30 13:59:16.033479 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:16.033442    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"]
Oct 30 13:59:16.045620 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:16.045582    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd9da67-cshsq"]
Oct 30 13:59:16.058607 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:16.058579    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mnrt8_c2c5cff9-185e-482f-8a61-b2ec954a3c14/discovery/0.log"
Oct 30 13:59:16.100398 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:16.100363    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgkz8_d33694bd-a804-4182-8d4b-e30156066d8f/istio-proxy/0.log"
Oct 30 13:59:16.101791 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:16.101766    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgkz8_d33694bd-a804-4182-8d4b-e30156066d8f/istio-proxy/1.log"
Oct 30 13:59:16.140661 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:16.140629    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-gvclh_81bb41a5-08e3-4e05-baee-c015d7239d96/istio-proxy/0.log"
Oct 30 13:59:16.245550 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:16.245515    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-czngj_d662a085-9c77-4c7c-82e1-de6c7101715d/istio-proxy/0.log"
Oct 30 13:59:16.365910 ip-10-0-135-24 kubenswrapper[4993]: I1030 13:59:16.365816    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77bb4759d7-ztvnj_51f4f667-c671-472c-ae68-beb10d17b645/router/0.log"